Skip to content

Commit cfc69de

Browse files
AbeJellinekAbeJellinek
committed
Add more detailed and hopefully useful agent skills
- Detailed instructions and sample code for each translator type - Scripts and associated skills for full end-to-end translator test creation, running, and updating - Scripts/skills for analyzing web pages via devtools and traffic monitoring (converts a HAR to Swagger using mitmproxy2swagger) - Various subsidiary improvements to .ci/ scripts for reliability and to extract shared constants
1 parent d26041b commit cfc69de

25 files changed

Lines changed: 2011 additions & 163 deletions

File tree

‎.agent/skills/capture-api/SKILL.md‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
---
2+
name: capture-api
3+
description: Analyze a website's API by capturing network traffic (HAR) and generating an OpenAPI spec via mitmproxy2swagger.
4+
---
5+
6+
## When to use
7+
8+
Most modern sites load bibliographic data via JavaScript API calls, not in the static HTML. Since translators receive a static DOM copy, you need to identify and replicate those API calls using `requestJSON()`, `requestText()`, or `requestDocument()`.
9+
10+
## Capture and analyze
11+
12+
```
13+
node .bin/capture-har.mjs "<url>"
14+
```
15+
16+
This captures all network traffic via headless Chrome and generates an OpenAPI spec using mitmproxy2swagger. The tool requires mitmproxy2swagger and will prompt to install it if missing (`uv tool install mitmproxy2swagger`).
17+
18+
The tool automatically runs two passes: first to discover all paths, then to generate full request/response schemas for API-like paths.
19+
20+
**Read the output YAML file.** It contains everything you need: endpoint paths, HTTP methods, query parameters, and complete response schemas with field names and types. Do NOT go back to the raw HAR file - the YAML is the authoritative, structured output. Use it directly to understand the API and write your translator.
21+
22+
If the tool reports "No obvious API paths found", edit the YAML to remove the `ignore:` prefix from paths you're interested in, then re-run the command.
23+
24+
Options:
25+
- `--interact`: Open a headed browser so you can click around the site before capturing.
26+
- `--no-openapi`: Skip mitmproxy2swagger and just save the raw HAR file.
27+
28+
## Use the discovered API in your translator
29+
30+
Once you identify the relevant API endpoints:
31+
32+
1. Determine how to construct the API URL from the page URL (e.g. extracting an article ID).
33+
2. Use `requestJSON()` to call the API directly in `scrape()`.
34+
3. Map the API response fields to Zotero item fields.
35+
36+
```js
37+
async function scrape(doc, url) {
38+
let articleId = url.match(/\/article\/(\d+)/)[1];
39+
let data = await requestJSON(`https://api.example.com/articles/${articleId}`);
40+
let item = new Zotero.Item('journalArticle');
41+
item.title = data.title;
42+
// ... map other fields ...
43+
item.complete();
44+
}
45+
```

‎.agent/skills/create-test/SKILL.md‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
---
2+
name: create-test
3+
description: Create or update test cases for a Zotero translator by running it against live URLs and capturing the output.
4+
---
5+
6+
## How test cases work
7+
8+
Test cases are embedded at the end of each translator file between `/** BEGIN TEST CASES **/` and `/** END TEST CASES **/`. They look like JS but **the content is strict JSON** — double-quoted keys, no trailing commas, no comments, no single quotes. The linter enforces this.
9+
10+
```js
11+
/** BEGIN TEST CASES **/
12+
var testCases = [
13+
{
14+
"type": "web",
15+
"url": "https://example.com/article/123",
16+
"items": [
17+
{
18+
"itemType": "journalArticle",
19+
"title": "Example Article",
20+
...
21+
}
22+
]
23+
}
24+
]
25+
/** END TEST CASES **/
26+
```
27+
28+
## Creating test cases
29+
30+
The tool runs the translator, captures its output, and writes the test case JSON directly into the file. Do NOT hand-write test case item objects.
31+
32+
### Web
33+
34+
```
35+
node .bin/create-test.mjs "<translator filename>" --url "<url>"
36+
```
37+
38+
### Search
39+
40+
```
41+
node .bin/create-test.mjs "<translator filename>" --search '{"DOI":"10.1234/example"}'
42+
```
43+
44+
### Import
45+
46+
```
47+
node .bin/create-test.mjs "<translator filename>" --input "<import data>"
48+
```
49+
50+
### Export
51+
52+
Export tests are not supported by the automated test runner. Test export translators manually.
53+
54+
## Options
55+
56+
- `--no-write`: Preview the captured test case without modifying the file.
57+
- `--json`: Output structured JSON.
58+
59+
## Guidelines
60+
61+
- Include at least one single-item test and one `"multiple"` test (if supported).
62+
- Use stable URLs unlikely to change (DOI-based, permalinks, archived content).
63+
- Always use this tool to generate test cases. Never write test case items by hand — the tool ensures the JSON matches what the translator actually produces.
64+
- If you need to update a test case after changing translator code, re-run the tool for that URL. Replace the old test case entry with the new output.

‎.agent/skills/develop-export-translator/SKILL.md‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
---
2+
name: develop-export-translator
3+
description: Develop an export translator that converts Zotero items into a file format (JSON, XML, CSV, etc.).
4+
---
5+
6+
## Prerequisites
7+
8+
Fetch and read the Zotero translator documentation:
9+
- https://www.zotero.org/support/_export/raw/dev/translators
10+
- https://www.zotero.org/support/_export/raw/dev/translators/coding
11+
12+
Also read `index.d.ts` for type definitions.
13+
14+
## Step 1: Gather information
15+
16+
1. **Label**: The format name (e.g. "My Custom JSON Export")
17+
2. **Creator**: The author's name
18+
3. **Expected output**: A sample of what the exported format should look like
19+
20+
Look for existing export translators for patterns:
21+
```
22+
grep -l "doExport" *.js
23+
```
24+
25+
## Step 2: Initialize
26+
27+
```
28+
node .bin/init-translator.mjs --label "<Label>" --creator "<Creator>" --type export
29+
```
30+
31+
If the translator should also import the same format, use `--type import,export` and implement `detectImport()` and `doImport()` as described in the `develop-import-translator` skill.
32+
33+
Export translators have no `target` regex.
34+
35+
## Step 3: Write the code
36+
37+
### `doExport()`
38+
39+
Loop through items with `Zotero.nextItem()` and write output with `Zotero.write()`.
40+
41+
```js
42+
function doExport() {
43+
let first = true;
44+
Zotero.write('[\n');
45+
let item;
46+
while ((item = Zotero.nextItem())) {
47+
if (!first) Zotero.write(',\n');
48+
first = false;
49+
Zotero.write(JSON.stringify({
50+
title: item.title,
51+
authors: item.creators.filter(c => c.creatorType === 'author')
52+
.map(c => `${c.firstName} ${c.lastName}`),
53+
date: item.date,
54+
doi: item.DOI,
55+
// ... map other fields ...
56+
}, null, '\t'));
57+
}
58+
Zotero.write('\n]\n');
59+
}
60+
```
61+
62+
Key APIs:
63+
- `Zotero.nextItem()` — returns the next item object, or `false` when done.
64+
- `Zotero.write(string)` — write to output.
65+
- `Zotero.getOption(name)` — read export options (configured via `displayOptions` in the header metadata).
66+
- `Zotero.nextCollection()` — iterate collections (requires `configOptions: { getCollections: true }` in header).
67+
68+
### Header options for export translators
69+
70+
The metadata header can include:
71+
72+
```json
73+
"configOptions": {
74+
"async": true,
75+
"getCollections": true
76+
},
77+
"displayOptions": {
78+
"exportNotes": true,
79+
"exportFileData": false
80+
}
81+
```
82+
83+
## Step 4: Create tests
84+
85+
Export tests are not yet supported by the automated test runner. Test manually by exporting items from Zotero and verifying the output format.
86+
87+
## Step 5: Verify and submit
88+
89+
**Update `lastUpdated` every time you modify translator code.**
90+
91+
```
92+
node .bin/update-metadata.mjs "<Label>.js"
93+
npm run lint -- "<Label>.js"
94+
```

‎.agent/skills/develop-import-translator/SKILL.md‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
---
2+
name: develop-import-translator
3+
description: Develop an import translator that parses a file format (JSON, XML, RIS, BibTeX, CSV, etc.) into Zotero items.
4+
---
5+
6+
## Prerequisites
7+
8+
Fetch and read the Zotero translator documentation:
9+
- https://www.zotero.org/support/_export/raw/dev/translators
10+
- https://www.zotero.org/support/_export/raw/dev/translators/coding
11+
12+
Also read `index.d.ts` for type definitions.
13+
14+
## Step 1: Gather information
15+
16+
1. **Label**: The format name (e.g. "My Custom JSON")
17+
2. **Creator**: The author's name
18+
3. **Example data**: A sample of the format to import
19+
20+
Look for existing import translators that handle similar formats:
21+
```
22+
grep -l "detectImport\|doImport" *.js
23+
```
24+
25+
## Step 2: Initialize
26+
27+
```
28+
node .bin/init-translator.mjs --label "<Label>" --creator "<Creator>" --type import
29+
```
30+
31+
If the translator should also export the same format, use `--type import,export` and implement `doExport()` as described in the `develop-export-translator` skill.
32+
33+
Import translators have no `target` regex — they match on content via `detectImport()`.
34+
35+
## Step 3: Write the code
36+
37+
### `detectImport()`
38+
39+
Read the first few lines with `Zotero.read()` and return `true` if the format matches. Be specific to avoid false positives with other formats.
40+
41+
```js
42+
function detectImport() {
43+
let start = '';
44+
for (let i = 0; i < 10; i++) {
45+
let line = Zotero.read();
46+
if (line === false) break;
47+
start += line;
48+
}
49+
// Check for a distinctive marker in the format
50+
return start.includes('"my_format_version"');
51+
}
52+
```
53+
54+
### `doImport()`
55+
56+
Read the full input, parse it, and create items.
57+
58+
```js
59+
async function doImport() {
60+
// Read all input
61+
let text = '';
62+
let line;
63+
while ((line = Zotero.read()) !== false) {
64+
text += line;
65+
}
66+
67+
let data = JSON.parse(text);
68+
for (let record of data.records) {
69+
let item = new Zotero.Item('journalArticle');
70+
item.title = record.title;
71+
item.date = record.date;
72+
for (let author of record.authors) {
73+
item.creators.push({
74+
firstName: author.first,
75+
lastName: author.last,
76+
creatorType: 'author',
77+
});
78+
}
79+
// ... map other fields ...
80+
item.complete();
81+
}
82+
}
83+
```
84+
85+
Key APIs:
86+
- `Zotero.read(length)` — read characters from input. Returns `false` at EOF. With no argument, reads one line.
87+
- `new Zotero.Item(itemType)` — create an item. Set fields as properties, then call `.complete()`.
88+
- For XML: `(new DOMParser()).parseFromString(text, 'text/xml')`
89+
- For line-based formats (RIS, BibTeX): read and parse line by line.
90+
91+
## Step 4: Create tests
92+
93+
```
94+
node .bin/create-test.mjs "<Label>.js" --input '<paste example data here>'
95+
```
96+
97+
## Step 5: Verify and submit
98+
99+
**Update `lastUpdated` every time you modify translator code.**
100+
101+
```
102+
node .bin/update-metadata.mjs "<Label>.js"
103+
npm run lint -- "<Label>.js"
104+
node .bin/run-tests.mjs "<Label>.js"
105+
```

0 commit comments

Comments
 (0)