Umitake AI OCR for Kintone — User Manual

Table of Contents

1. Overview
2. Installation
3. Getting API Keys
   • Anthropic (Claude)
   • OpenAI (GPT)
   • Azure OpenAI
   • Google Gemini
4. Plugin Configuration
   • License Key
   • AI Provider and Model
   • Common Prompt
   • Pattern A — Record-level OCR
   • Pattern B — Table Row-level OCR
5. Using Pattern A (Record-level OCR)
6. Using Pattern B (Table Row-level OCR)
7. AI Lookup Search
8. Notes and Limitations
9. Troubleshooting

Overview

The AI OCR Plugin reads PDF and image files (JPG, PNG, WebP) using AI and automatically fills kintone record fields with the extracted values.

Key features:

• Extract field values from invoices, forms, receipts, and any other document
• Support for four AI providers: Claude (Anthropic), OpenAI (GPT), Azure OpenAI, and Google Gemini
• Two operating modes:
  • Pattern A — Record-level OCR with a side-by-side review screen before applying values
  • Pattern B — Table row-level OCR with a per-row review screen
• AI Lookup Search — AI automatically selects the best matching master record for lookup fields
• Automatically save the OCR source file to an attachment field

Installation

1. Open the kintone admin console (gear icon in the top right)
2. Click Plugins
3. Click Import Plugin
4. Select and upload kintone-ai-ocr.zip
5. After uploading, open the target app's settings
6. From the Plugins tab, add "AI OCR"
7. Click Settings to open the plugin configuration screen

Getting API Keys

To use the AI OCR Plugin, you need an API key from your chosen AI provider. The API key is entered in the plugin settings and is stored encrypted in kintone's plugin configuration.

Anthropic (Claude)

Claude provides native PDF support with high accuracy, especially for scanned documents.

1. Go to https://platform.claude.com/ and sign in (or create an account)
2. From the left menu, click API Keys
3. Click Create Key, enter a name, and click Create Key
4. Copy the key that starts with sk-ant- — it is shown only once
5. Paste the key into the API Key field in the plugin settings

Note: Anthropic charges based on token usage. You can check your usage and credits at https://platform.claude.com/settings/billing.

OpenAI (GPT)

1. Go to https://platform.openai.com/ and sign in (or create an account)
2. From the top-right menu, click your organization name → API keys
3. Click Create new secret key, enter a name, and click Create secret key
4. Copy the key that starts with sk- — it is shown only once
5. Paste the key into the API Key field in the plugin settings

Note: PDFs are converted to images (300 DPI PNG) using PDF.js before being sent to the API. Digital PDFs are highly accurate, but scanned PDFs may have slightly reduced accuracy compared to Claude.

Azure OpenAI

Azure OpenAI requires an active Azure subscription and an approved Azure OpenAI resource.

Step 1 — Create an Azure OpenAI resource:

1. Sign in to https://portal.azure.com/
2. Search for Azure OpenAI and click Create
3. Fill in subscription, resource group, region, and resource name, then click Create

Step 2 — Deploy a model:

1. Open the created resource and click Go to Azure OpenAI Studio (or navigate to https://oai.azure.com/)
2. Click Deployments → Create new deployment
3. Select a model (e.g., gpt-4o) and enter a deployment name (e.g., gpt-4o-deployment)
4. Note the deployment name — you will need it in the plugin settings

Step 3 — Get the API key and endpoint:

1. In the Azure portal, open your Azure OpenAI resource
2. Click Keys and Endpoint (under Resource Management)
3. Copy KEY 1 (or KEY 2)
4. Note the Resource Name from the endpoint URL (e.g., https://<resource-name>.openai.azure.com/)

Step 4 — Enter in plugin settings:

Google Gemini

Option A — Google AI Studio (simple, for personal use):

1. Go to https://aistudio.google.com/ and sign in with a Google account
2. Click Get API key → Create API key
3. Copy the generated key
4. Paste the key into the API Key field in the plugin settings

Option B — Google Cloud (for production / organization use):

1. Sign in to https://console.cloud.google.com/
2. Create or select a project
3. Enable the Generative Language API (search for it in the API Library)
4. Navigate to APIs & Services → Credentials → Create Credentials → API key
5. Copy the generated key and paste it into the plugin settings

Plugin Configuration

Click Settings for "AI OCR" in your app's plugin settings to open the configuration screen.

License Key

If you have a license key, enter it in the License Key field and click Verify. A license is required to use this plugin.

AI Provider and Model

1. Select your AI provider from the AI Provider dropdown
2. For Anthropic, the available models are listed automatically
3. For OpenAI and Gemini, click Load Models to fetch the available models from the API
4. For Azure OpenAI, enter the Deployment Name directly (this is the name you chose when deploying the model in Azure OpenAI Studio)
5. Additional Azure fields:
   • Azure Resource Name — the resource name from your endpoint URL
   • API Version — e.g., 2024-08-01-preview

PDF support by provider:

Common Prompt

Enter general extraction instructions that apply to all OCR runs. This is sent to the AI every time OCR is executed.

Example:

Extract the following fields from the invoice. Return only the values explicitly written in the document. If a field is not found, return an empty string.

Pattern A — Record-level OCR

Enable Pattern A to add an OCR widget to a record's edit screen.

Additional Prompt Field — Usage Examples

The Additional Prompt Field lets you give the AI different OCR instructions on a per-record basis. The content of the specified multi-line text field is appended to the Common Prompt at OCR execution time.

Handling special document formats

Use it to describe per-document reading rules that differ from the norm:

This invoice shows tax-inclusive amounts only.
Calculate the pre-tax amount as: amount ÷ 1.1.

Extract only the product code in the format: 2 letters + 4 digits (e.g., AB-0123).

Providing hints for lookup matching

When the name on a document doesn't exactly match a kintone master record, combine this with AI Lookup Search to improve accuracy:

If the vendor name contains abbreviations like "Inc." or "Co.", match against the full
company name in the master.

Copying rules from a master record via Lookup

You can store OCR instructions in a "Document Type Master" kintone app and copy them into this field using a Lookup field. This lets you manage per-vendor or per-document-type rules centrally without requiring operators to type instructions each time.

Field Mappings

Maps OCR result keys to kintone fields.

Click + Add Field to add a row. Click × to remove.

Example: To extract "invoice number" into a kintone field with field code invoice_no:

The AI's system prompt instructs it to return JSON like { "invoice_number": "INV-001", ... }. The plugin maps invoice_number → invoice_no.

Table Mappings

Maps OCR result table rows to a kintone subtable field. Use this to populate line-item tables from invoices, etc.

Multiple table mappings can be added for different subtables.

Pattern B — Table Row-level OCR

Enable Pattern B to replace a kintone subtable with a custom table that has an OCR button on each row.

Column Mappings

Using Pattern A (Record-level OCR)

1. Open a record in Edit mode
2. The OCR widget is displayed in the Space field you configured
3. Select a file: drag and drop a PDF or image onto the drop zone, or click to browse
4. (Optional) Enter additional instructions in the Additional prompt text area
5. Click Run OCR
6. The plugin sends the file to the AI and displays the OCR Review modal when the result is ready
7. In the review modal:
   • The document preview is shown on the left (multi-page PDFs show page navigation arrows)
   • The extracted field values are shown on the right — you can edit them before applying
   • For table mappings, extracted rows are shown in a table section at the bottom
8. Click Apply to Record to fill the kintone fields with the reviewed values
9. Save the record normally — if an attachment field is configured, the source file is uploaded at this point

Using Pattern B (Table Row-level OCR)

1. Open a record in Edit mode
2. The custom OCR table is displayed in the Space field you configured
3. Each row has an 📎 OCR button in the action column
4. Click 📎 OCR on the row you want to fill
5. A file browser opens — select a PDF or image
6. The plugin runs OCR and displays the Row OCR Result modal
7. Review and edit the extracted values, then click Apply to Row
8. Repeat for each row as needed
9. Use the ＋ Add Row button to add new rows, or × to remove rows
10. Save the record normally — attachment files are uploaded at save time

AI Lookup Search

When a field mapped to an OCR result key is a lookup field, you can enable AI Lookup Search to have the AI automatically select the best matching master record.

How it works

1. Before running OCR, the plugin fetches all records from the lookup's source app and caches them
2. After OCR extracts field values, the AI compares the OCR result against the candidate records
3. The AI returns the key value of the best matching record
4. The matched value is pre-filled in the review modal

Enabling AI Lookup Search

In Pattern A — Field Mappings:

When you select a kintone lookup field in the Kintone Field dropdown, an AI Lookup Search checkbox appears below the row. Check it to enable AI matching for that field.

In Pattern B — Column Mappings:

When you select a lookup column in the Column dropdown, an AI Lookup Search checkbox appears. Check it to enable.

Use case examples

• Vendor name: OCR reads "ABC Trading Co." from an invoice → AI searches the vendor master and fills in the correct vendor code
• Account code: OCR reads an expense description → AI selects the appropriate account code from the chart of accounts
• Product code: OCR reads a product name → AI matches it to the item master and fills in the item code

Note: AI Lookup Search makes additional API calls to the AI. If no good match is found, the raw OCR text is used instead.

Notes and Limitations

Supported file types

PDF, JPG, JPEG, PNG, WebP

API key security

API keys are stored in kintone's plugin configuration (server-side encrypted storage). They are never sent to any server other than the AI provider's API endpoint via kintone.proxy.

PDF accuracy

For high-volume scanned document processing, Claude (Anthropic) is recommended.

Pattern A and Pattern B cannot share the same Space field

If both patterns are enabled, each must be assigned a different Space field. The plugin settings screen shows a warning if the same Space field is selected for both.

kintone subtable Space field limitation

kintone does not allow Space fields inside subtable rows. This is why Pattern B replaces the entire subtable with a custom table rendered in a separate Space field.

Display language

The OCR widget and plugin settings screen automatically display in Japanese or English based on the logged-in user's language setting (kintone.getLoginUser().language).

Troubleshooting

The OCR widget does not appear

Check:

• Plugin settings have been saved (did you click Save on the settings screen?)
• Pattern A (or B) is enabled in the plugin settings
• The Space field's Element ID in the kintone form settings matches the Space Element ID in the plugin settings
• The app settings have been saved and the app has been updated

OCR runs but fields are not filled

Check:

• The OCR Result Key in Field Mappings exactly matches the key name you used in the Common Prompt (the AI must output JSON with this exact key)
• The Kintone Field selected in the mapping is correct
• Click Apply to Record (or Apply to Row) in the review modal — values are not applied automatically

AI API error

Check:

• The API key is correct and has not expired
• The model selected in the plugin settings is available (some older models may be deprecated)
• For Azure OpenAI: the Resource Name, API Version, and Deployment Name are all correct
• Your AI account has sufficient credits/quota

License error

Check:

• The license key is entered in the plugin settings and has been verified
• The license has not expired

Lookup field is not recognized as a lookup (AI Lookup Search checkbox does not appear)

Check:

• The field selected in the mapping is actually a lookup field in the kintone form settings
• The app fields were loaded correctly (try refreshing the plugin settings screen)

For Pattern B: Changes in the OCR table are not reflected when saving

Check:

• Do not edit the original kintone subtable directly — always use the custom OCR table in the Space field
• If the original subtable is visible, contact your kintone administrator to verify the Space field placement