Umitake AI OCR for Kintone — User Manual

Table of Contents

1. Overview
2. Installation
3. Getting API Keys
   • Anthropic (Claude)
   • OpenAI (GPT)
   • Azure OpenAI
   • Google Gemini
4. Plugin Configuration
   • License Key
   • AI Provider and Model
   • Common Prompt
   • Pattern A — Record-level OCR
   • Pattern B — Table Row-level OCR
5. Using Pattern A (Record-level OCR)
6. Using Pattern B (Table Row-level OCR)
   • Processing one row at a time
   • Processing many files at once (bulk OCR)
   • Row status
7. Bulk import (one record per file)
8. AI Lookup Search
9. Notes and Limitations
10. Troubleshooting

Overview

AI OCR reads PDF, image, and text files (JPG, PNG, WebP, TXT, CSV, JSON) using AI and automatically fills kintone record fields with the extracted values.

Key features:

• Extract field values from invoices, forms, receipts, and any other document
• Analyze the contents of text files (TXT, CSV, JSON) with AI and populate fields
• Support for four AI providers: Claude (Anthropic), OpenAI (GPT), Azure OpenAI, and Google Gemini
• Two operating modes:
  • Pattern A — Record-level OCR with a side-by-side review screen before applying values
  • Pattern B — Table row-level OCR with a per-row review screen
• Bulk OCR (Pattern B) — drop in multiple files at once and let the plugin process them 3 at a time in parallel
• Bulk import (Pattern A) — register one record per file from the list screen, then run OCR on the whole view at once
• Skip review option — apply OCR results directly without opening the review modal, and reopen it later if needed
• AI Lookup Search — AI automatically selects the best matching master record for lookup fields
• Automatically save the OCR source file to an attachment field

Installation

1. Open the kintone admin console (gear icon in the top right)
2. Click Plugins
3. Click Import Plugin
4. Select and upload plugin.zip
5. After uploading, open the target app's settings
6. From the Plugins tab, add "Umitake AI OCR for Kintone"
7. Click Settings to open the plugin configuration screen

Getting API Keys

To use Umitake AI OCR, you need an API key from your chosen AI provider. The API key is entered in the plugin settings and is stored encrypted in kintone's plugin configuration.

Anthropic (Claude)

Claude provides native PDF support with high accuracy, especially for scanned documents.

1. Go to https://platform.claude.com/ and sign in (or create an account)
2. From the left menu, click API Keys
3. Click Create Key, enter a name, and click Create Key
4. Copy the key that starts with sk-ant- — it is shown only once
5. Paste the key into the API Key field in the plugin settings

Note: Anthropic charges based on token usage. You can check your usage and credits at https://platform.claude.com/settings/billing. A link is also provided directly in the plugin settings screen.

OpenAI (GPT)

1. Go to https://platform.openai.com/ and sign in (or create an account)
2. From the top-right menu, click your organization name → API keys
3. Click Create new secret key, enter a name, and click Create secret key
4. Copy the key that starts with sk- — it is shown only once
5. Paste the key into the API Key field in the plugin settings

Note: PDFs are converted to images (JPEG) using PDF.js before being sent to the API. Digital PDFs are highly accurate, but scanned PDFs may have slightly reduced accuracy compared to Claude.

Azure OpenAI

Azure OpenAI requires an active Azure subscription and an approved Azure OpenAI resource.

Step 1 — Create an Azure OpenAI resource:

1. Sign in to https://portal.azure.com/
2. Search for Azure OpenAI and click Create
3. Fill in subscription, resource group, region, and resource name, then click Create

Step 2 — Deploy a model:

1. Open the created resource and click Go to Microsoft Foundry portal (formerly "Azure OpenAI Studio"; navigate to https://ai.azure.com/, or the older https://oai.azure.com/ which still redirects there)
2. Click Deployments → Create new deployment
3. Select a model (e.g., gpt-4o) and enter a deployment name (e.g., gpt-4o-deployment)
4. Note the deployment name — you will need it in the plugin settings

Step 3 — Get the API key and endpoint:

1. In the Azure portal, open your Azure OpenAI resource
2. Click Keys and Endpoint (under Resource Management)
3. Copy KEY 1 (or KEY 2)
4. Note the Resource Name from the endpoint URL (e.g., https://<resource-name>.openai.azure.com/)

Step 4 — Enter in plugin settings:

Google Gemini

Option A — Google AI Studio (simple, for personal use):

1. Go to https://aistudio.google.com/ and sign in with a Google account
2. Click Get API key → Create API key
3. Copy the generated key
4. Paste the key into the API Key field in the plugin settings

Option B — Google Cloud (for production / organization use):

1. Sign in to https://console.cloud.google.com/
2. Create or select a project
3. Enable the Generative Language API (search for it in the API Library)
4. Navigate to APIs & Services → Credentials → Create Credentials → API key
5. Copy the generated key and paste it into the plugin settings

Note: Google has been phasing out old-style "standard" API keys in favor of "auth keys." Keys created in AI Studio going forward are auth keys automatically. If OCR calls to Gemini start failing with an authentication error, generate a new key from the link above and update it in the plugin settings.

Plugin Configuration

Click Settings for "Umitake AI OCR for Kintone" in your app's plugin settings to open the configuration screen.

License Key

If you have a license key, enter it in the License Key field and click Verify. A license is required to use this plugin.

AI Provider and Model

1. Select your AI provider from the AI Provider dropdown
2. For Anthropic, the available models are listed automatically
3. For OpenAI and Gemini, click Load Models to fetch the available models from the API
4. For Azure OpenAI, enter the Deployment Name directly (this is the name you chose when deploying the model in the Microsoft Foundry portal (formerly Azure OpenAI Studio))
5. Additional Azure fields:
   • Azure Resource Name — the resource name from your endpoint URL
   • API Version — e.g., 2024-08-01-preview

PDF support by provider:

PDF image resolution (shown only when OpenAI / Azure OpenAI is selected)

Controls the resolution used when converting PDF pages to images.

Claude and Gemini send the PDF as-is, so this setting is not shown for them.

Common Prompt

Enter general extraction instructions that apply to all OCR runs. This is sent to the AI every time OCR is executed.

Example:

Extract the following fields from the invoice. Return only the values explicitly written in the document. If a field is not found, return an empty string.

Pattern A — Record-level OCR

Enable Pattern A to add an OCR widget to a record's edit screen.

Field Mappings

Maps OCR result keys to kintone fields.

Click + Add Field to add a row. Click × to remove.

Example: To extract "invoice number" into a kintone field with field code invoice_no:

The AI's system prompt instructs it to return JSON like { "invoice_number": "INV-001", ... }. The plugin maps invoice_number → invoice_no.

Table Mappings

Maps OCR result table rows to a kintone subtable field. Use this to populate line-item tables from invoices, etc.

AI Lookup Search: When you select a lookup column in the column mapping, an AI Lookup Search checkbox appears. Enable it to have the AI automatically select the best matching master record.

Editability on apply: A dropdown shown on every column. It sets kintone's disabled on the cell at the same time the OCR value is written into it.

Note: The editability lasts only while the record stays open. After saving and reopening the record, the column returns to its native kintone state — combine it with kintone's field permissions if you need permanent protection. Also avoid Make read-only on required columns that the OCR may leave empty, as they could no longer be filled in.

Multiple table mappings can be added for different subtables.

Pattern B — Table Row-level OCR

Enable Pattern B to replace a kintone subtable with a custom table that has an OCR button on each row.

Column Mappings

Maps each subtable column to an OCR result key.

AI Lookup Search: When you select a lookup column, an AI Lookup Search checkbox appears. Enable it to have the AI automatically select the best matching master record.

Editability on apply: Choose Leave unchanged (default), Make read-only, or Make editable. With Make read-only, the custom table cell becomes read-only as soon as an OCR result lands in that row, preventing manual edits. To correct a value, open the review screen with the 🔍 button.

Using Pattern A (Record-level OCR)

1. Open a record in Edit mode
2. The OCR widget is displayed in the Space field you configured
3. Select a file: drag and drop a file onto the drop zone, or click to browse
   • Supported formats: PDF / JPG / PNG / WebP / TXT / CSV / JSON
   • TXT, CSV, and JSON files must be 50 KB or less
4. (Optional) Enter additional instructions in the Additional prompt text area — for example, "The 'notes' field should be left blank" or "This is a receipt from vendor XYZ, treat the date format as MM/DD/YYYY"
5. Click Run OCR
   • With Skip review and apply directly checked, results go straight to the record without opening the modal. You can still inspect and correct them later with the Open review button
6. The plugin sends the file to the AI and displays the OCR Review modal when the result is ready
7. In the review modal:
   • For PDF/image files: the document preview is shown on the left (multi-page PDFs show page navigation arrows)
   • For text files: no preview is shown — only the extracted field values are displayed
   • The extracted field values are shown on the right — you can edit them before applying
   • For table mappings, extracted rows are shown in a table section at the bottom
8. Click Apply to Record to fill the kintone fields with the reviewed values
9. Save the record normally — if an attachment field is configured, the source file is uploaded at this point

Note: The review screen can be reopened after applying, via the Open review button. Reloading the page releases the retained file, after which it can no longer be opened.

Using Pattern B (Table Row-level OCR)

Processing one row at a time

1. Open a record in Edit mode
2. The custom OCR table is displayed in the Space field you configured
3. Each row has an 📎 OCR button in the action column
4. Click 📎 OCR on the row you want to fill
5. A file browser opens — select a file
   • Supported formats: PDF / JPG / PNG / WebP / TXT / CSV / JSON
   • TXT, CSV, and JSON files must be 50 KB or less
6. The plugin runs OCR and displays the Row OCR Result modal
   • For text files, no preview is shown (compact display)
   • If Skip review and apply directly is checked, the modal is not opened and the values are filled in immediately
7. Review and edit the extracted values, then click Apply to Row
8. Repeat for each row as needed
9. Use the ＋ button (to the left of × on each row) to insert a new row immediately after that row; use × to delete the row
10. Save the record normally — attachment files are uploaded at save time

Processing many files at once (bulk OCR)

You do not have to process documents one by one.

1. Drag and drop several files onto the bulk OCR area above the table (or click it to select multiple files)
2. A dialog shows how many files will be processed — click OK
   • Up to 20 files per batch
   • Unsupported or oversized files are skipped automatically and the count is reported
3. Rows are allocated automatically and processed 3 at a time in parallel
4. Each row fills in as its file finishes and is marked Unconfirmed
   • Progress is shown as "3 / 10 completed"
   • Click Cancel remaining to stop — files already being processed will finish
   • If one file fails, the rest continue. The failed row shows Error and can be retried with its own 📎 OCR button
5. Click the 🔍 button on any row to open the review screen for that row
6. Review and click Apply to Row — the Unconfirmed badge clears
7. Save the record normally — each row gets its own attached file

Note: The bulk OCR area can be hidden with Show the bulk OCR upload area in the plugin settings.

Row status

Note: Multi-line text (MULTI_LINE_TEXT) fields are displayed as textareas both in the table and in the review screen, so line breaks are preserved.

Note: The review screen can be reopened at any time. Even after Apply to Row, or when review was skipped, the 🔍 button opens it again. Reloading the page releases the retained file, after which it can no longer be opened.

Bulk import (one record per file)

You do not have to register documents one at a time. The flow is split into two stages.

These panels appear only when Show bulk import on the record list screen is enabled in the Pattern A plugin settings and an Attachment Field is configured.

Stage 1 — Register files (PC and mobile)

This works from mobile too. No AI is called in this stage, so it costs nothing.

1. Open the record list screen
2. Drop files onto the "Bulk import files" area, or click/tap to select them
   • On mobile you can also take a photo on the spot
   • Supported formats: PDF / JPG / PNG / WebP / TXT / CSV / JSON
   • Up to 100 files per import
3. Click Create N record(s)
4. One record is created per file, with that file in the record's attachment field
5. The list reloads when it finishes

Note: Records are created in batches of 100. If a batch fails, none of its records are created, so nothing half-finished is left behind and you can simply run it again.

Stage 2 — Run bulk OCR (PC only)

1. Filter the list view down to the records you want to OCR
   • A dedicated "not yet OCR'd" view (e.g. key fields empty and an attachment present) makes this easy
   • Records without an attachment are skipped automatically
   • Up to the first 100 records are processed per run
2. Optionally tick Review results before applying (off by default)
3. Click Run OCR on this view
4. Confirm the record count in the dialog
5. Records are processed 3 at a time; progress shows as "3 / 10 completed"
   • Unticked: each record is updated as soon as its OCR finishes
   • Ticked: after all records finish, the review screen opens one record at a time. "Apply to Record" applies and moves on; "Cancel" skips that record
6. A summary shows how many succeeded and failed. Failed records are left untouched, so you can simply run it again

Notes

• HEIC (the iPhone default) is not supported. Set Camera > Formats to "Most Compatible" so photos are taken as JPEG
• Stage 2 runs only on the PC list screen (it is not shown on mobile)

AI Lookup Search

When a field mapped to an OCR result key is a lookup field, you can enable AI Lookup Search to have the AI automatically select the best matching master record.

How it works

1. Before running OCR, the plugin fetches all records from the lookup's source app and caches them (the cache is kept for 5 minutes and is not re-fetched during that time)
2. After OCR extracts field values, the AI compares the OCR result against the candidate records
   • If the source app has 300 records or fewer, all of them are used as candidates
   • Above 300 records, candidates are narrowed by similarity to the OCR text before being sent to the AI
3. The AI returns the key value of the best matching record
4. The matched value is pre-filled in the review modal

Note: A record added to the source app may not be matched for up to 5 minutes while the cache is still valid. Reload the record screen to pick it up immediately.

Enabling AI Lookup Search

In Pattern A — Field Mappings:

When you select a kintone lookup field in the Kintone Field dropdown, an AI Lookup Search checkbox appears below the row. Check it to enable AI matching for that field.

In Pattern B — Column Mappings:

When you select a lookup column in the Column dropdown, an AI Lookup Search checkbox appears. Check it to enable.

Use case examples

• Vendor name: OCR reads "ABC Trading Co." from an invoice → AI searches the vendor master and fills in the correct vendor code
• Account code: OCR reads an expense description → AI selects the appropriate account code from the chart of accounts
• Product code: OCR reads a product name → AI matches it to the item master and fills in the item code

Note: AI Lookup Search makes additional API calls to the AI. If no good match is found, the raw OCR text is used instead.

Notes and Limitations

Supported file types

CSV files are formatted with a header line and data rows so the AI understands the column structure.

Text file review modals show no document preview (compact field-only display).

API key security

API keys are stored in kintone's plugin configuration (server-side encrypted storage). They are never sent to any server other than the AI provider's API endpoint via kintone.proxy.

PDF accuracy

For high-volume scanned document processing, Claude (Anthropic) is recommended.

If accuracy is insufficient for scanned documents on OpenAI / Azure OpenAI, change PDF image resolution in the plugin settings to "Standard (200 DPI)" or "High accuracy (300 DPI)".

Additional Prompt Field vs. Additional prompt textarea

• Additional Prompt Field (configured in plugin settings): A kintone field (multi-line text) whose stored value is automatically included in every OCR prompt for that app. Useful for persistent, per-app or per-vendor instructions stored in a master record.
• Additional prompt textarea (shown in the OCR widget): A free-text input the user fills in at the time of OCR. Useful for one-off instructions that vary per scan.

Both can be used simultaneously — the field value is added first, then the textarea value.

Pattern A and Pattern B cannot share the same Space field

If both patterns are enabled, each must be assigned a different Space field. The plugin settings screen shows a warning if the same Space field is selected for both.

kintone subtable Space field limitation

kintone does not allow Space fields inside subtable rows. This is why Pattern B replaces the entire subtable with a custom table rendered in a separate Space field.

Display language

The OCR widget and plugin settings screen automatically display in Japanese or English based on the logged-in user's language setting (kintone.getLoginUser().language).

Troubleshooting

The OCR widget does not appear

Check:

• Plugin settings have been saved (did you click Save on the settings screen?)
• Pattern A (or B) is enabled in the plugin settings
• The Space field's Element ID in the kintone form settings matches the Space Element ID in the plugin settings
• The app settings have been saved and the app has been updated

OCR runs but fields are not filled

Check:

• The OCR Result Key in Field Mappings exactly matches the key name you used in the Common Prompt (the AI must output JSON with this exact key)
• The Kintone Field selected in the mapping is correct
• Click Apply to Record (or Apply to Row) in the review modal — values are not applied automatically

AI API error

Check:

• The API key is correct and has not expired
• The model selected in the plugin settings is available (some older models may be deprecated)
• For Azure OpenAI: the Resource Name, API Version, and Deployment Name are all correct
• Your AI account has sufficient credits/quota

License error

Check:

• The license key is entered in the plugin settings and has been verified
• The license has not expired

Lookup field is not recognized as a lookup (AI Lookup Search checkbox does not appear)

Check:

• The field selected in the mapping is actually a lookup field in the kintone form settings
• The app fields were loaded correctly (try refreshing the plugin settings screen)

For Pattern B: Changes in the OCR table are not reflected when saving

Check:

• Do not edit the original kintone subtable directly — always use the custom OCR table in the Space field
• If the original subtable is visible, contact your kintone administrator to verify the Space field placement