Repository navigation
Expand file tree
/
Copy pathgen_schema.py
More file actions
executable file
·2903 lines (2642 loc) · 131 KB
/
Copy pathgen_schema.py
File metadata and controls
executable file
·2903 lines (2642 loc) · 131 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
#!/usr/bin/env python3
# csnippets (c) 2019-2026 He Xian <hexian000@outlook.com>
# This code is licensed under MIT license (see LICENSE for details)
"""scripts/gen_schema.py
Generate C code from JSON Schema files: perfect-hash key dispatchers,
struct definitions, and marshal/unmarshal/free functions that use the
codec/json.h API.
Usage:
python3 scripts/gen_schema.py [OPTIONS] <schema.json> ...
Options:
--generate FEATURES Comma-separated list of features to generate:
structs — struct type definitions + free functions
unmarshal — lookup tables (always internal) +
unmarshal functions
marshal — marshal functions
lookup — expose the enum and lookup function
in the generated header (unmarshal
generates them internally regardless)
Default: structs,unmarshal,marshal.
Dependency rules:
unmarshal/marshal each imply structs; lookup does
not (it emits only the key enums and lookup
functions).
lookup tables are always generated internally for
unmarshal; lookup only controls whether the enum
and lookup function are made public.
Free functions are always generated with structs.
--prefix S Prepend S to every public generated symbol:
struct types, function names, enum names,
lookup functions, and enum-value prefixes.
Joined with a single '_' whether or not S ends in
one: "ex" and "ex_" both give struct ex_<stem>,
ex_unmarshal_<stem>, EX_<STEM>_<KEY>.
--optimize MODE Selects the code-generation backend and key-lookup
strategy together.
fast (default): bespoke per-scope unmarshal/marshal
functions + gperf perfect-hash lookup. Direct
dispatch; larger object code; needs gperf.
size: compact descriptor tables interpreted by the
generic runtime in json.h + memcmp binary-search
lookup. Object code shrinks dramatically (the
per-scope logic lives once in the runtime) at the
cost of table interpretation; no gperf needed.
Keys are emitted as quoted literals, so gperf
handles almost arbitrary bytes; an empty key, or
a key containing '?' (both of which gperf cannot
take), falls back to binary search.
NOTE: both modes require the shared runtime
(json_unmarshal/json_marshal/json_free) in codec/json.
--presence Track actual input properties with named bitfields.
--no-validate Omit schema-constraint validation from unmarshal.
--sub-schema Also make the free/unmarshal/marshal functions of
nested sub-schemas public (by default only the
root object's are; nested helpers are static).
--include-prefix P Include project headers as "P/json.h" rather than
"json.h" (default: codec).
--output-name BASE Name the generated files BASE.gen.{h,c} instead of
after the schema stem (sanitized to a C identifier).
Generated symbols still derive from the schema stem.
--indent STYLE Indentation of the generated C code: 'tab'
(default) for one tab per level, or an integer N
for N spaces per level.
Semantics notes:
* Fixed-property duplicates are last-value-wins; maps retain ordered
duplicate entries. Map keys and string values carry byte lengths,
including embedded NUL. Unconstrained map numbers retain their text.
* Missing properties receive complete defaults/type zeros and marshal
emits every property. Defaults do not satisfy required input checks;
zero fallbacks need not satisfy schema constraints when re-encoded.
* --presence instead emits has_<field> bitfields. Absent member storage
is untouched; marshal/free access only flagged members. Caller-built
output must initialize each value before setting its flag.
* Input strings/fragments borrow the mutable input buffer. Generated
json_defaults backing and array/map entries are owned per instance;
free every successful parse, even one containing only scalar fields.
* Map object/array values stay complete, syntax-checked raw fragments.
Their children/deep schema are not decoded or validated; the caller
explicitly parses them later. Fixed nested objects and object arrays
retain generated recursive parsing.
* additionalProperties is per object: false rejects extras, a supported
schema checks extras, omission/true allows them. Fixed objects discard
accepted extras.
* Extended codec functions take optional json_options for map filters and
one synchronous first-error callback with borrowed structured paths and
root-relative byte offsets. Output filters must be stable across calls.
* Marshal follows snprintf semantics and rejects non-finite doubles.
indent selects pretty output; NULL selects compact output.
* minLength/maxLength measure UTF-8 bytes. Supported scalar constraints
include bounds, multipleOf, const and enum; arrays support item checks
and minItems/maxItems. Unsupported refs, unions, boolean subschemas,
patterns and nested-array schemas retain raw-fragment fallback; other
unsupported validation keywords are omitted without warnings. Invalid
keyword values and unrepresentable supported defaults remain errors.
* Generated layouts/descriptors changed: regenerate with the matching
runtime. Maps replace former raw object members with typed entries.
Output per input schema (<stem> is the schema basename, or --output-name):
<dir>/<stem>.gen.h -- declarations (enums, lookups, structs, protos)
<dir>/<stem>.gen.c -- implementations (lookups, free, unmarshal, marshal)
Examples:
# generate all (default)
python3 scripts/gen_schema.py src/mux/proto_schema.json
# structs + unmarshal only, with a prefix to avoid name conflicts
python3 scripts/gen_schema.py --generate structs,unmarshal --prefix json_ \\
src/conf_schema.json
# full codec generation with prefix
python3 scripts/gen_schema.py --prefix json_ src/conf_schema.json
"""
import argparse
import json
import math
import re
import subprocess
import sys
from decimal import Decimal
from pathlib import Path
# ---------------------------------------------------------------------------
# Module constants
# ---------------------------------------------------------------------------
_INT32_MIN = -(2**31)
_INT32_MAX = 2**31 - 1
_UINT32_MAX = 2**32 - 1
_INT64_MIN = -(2**63)
_INT64_MAX = 2**63 - 1
_UINT64_MAX = 2**64 - 1
def _c_u64_literal(value: int) -> str:
"""Return a portable C literal for an unsigned 64-bit value."""
return f"UINT64_C({value})"
# Inclusive value range of each inferred C field type, used to drop a bound
# comparison that is always true/false for the type (e.g. `x < INT_MIN`), which
# would otherwise trip -Wtype-limits under -Werror. Unknown types get an
# unbounded range so every check is kept.
_C_TYPE_RANGE = {
"int": (_INT32_MIN, _INT32_MAX),
"unsigned": (0, _UINT32_MAX),
"intmax_t": (_INT64_MIN, _INT64_MAX),
"uintmax_t": (0, _UINT64_MAX),
}
# The complete --generate feature set; anything else on that option is a typo.
_FEATURES = frozenset({"structs", "unmarshal", "marshal", "lookup"})
# One indentation level for generated C source (default tab; set by --indent).
_INDENT = "\t"
_PRESENCE = False
class _JsonNumber(float):
"""Retain legal number spelling for untyped defaults and raw fragments.
Schema-directed arithmetic still uses float semantics; supported C doubles
are range-checked separately. Instance data must not lose digits or turn
a large exponent into an Infinity token during default serialization.
"""
def __new__(cls, text):
number = super().__new__(cls, text)
number.text = text
return number
def _default_spelling(value) -> str:
if isinstance(value, _JsonNumber):
return value.text
if isinstance(value, dict):
return "{" + ",".join(json.dumps(key, ensure_ascii=False) + ":" +
_default_spelling(item) for key, item in value.items()) + "}"
if isinstance(value, list):
return "[" + ",".join(_default_spelling(item) for item in value) + "]"
return json.dumps(value, ensure_ascii=False, allow_nan=False)
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def to_c_ident(s: str) -> str:
"""Return *s* sanitized into a valid, non-reserved C identifier."""
# Collapse each maximal run of non-identifier bytes to a single '_'. '_' is
# excluded from the retained class, so a literal '_' beside a separator (or a
# run of separators) folds in rather than forming an internal '__', which
# the generated names never contain -- a house rule kept stricter than
# ISO C11 §7.1.3, which reserves only a *leading* '__' (C++ [lex.name]
# reserves one anywhere); a lone '_' is a run of one and is kept.
ident = re.sub(r"[^A-Za-z0-9]+", "_", s)
# A name that does not start with a letter needs repairing, and the repair
# must not be an underscore: ISO C11 §7.1.3 reserves every identifier
# beginning with one for file scope -- where the schema prefix seeds the
# struct/enum tags and functions -- and reserves '_' + uppercase for any
# use. A leading letter keeps the result out of both namespaces. The same
# substitution can leave
# a leading '_' without any digit involved ('.env' -> '_env'), so key off
# "not a letter", not "digit". The empty string is a legal JSON key and
# maps to "x"; _make_field_map still resolves any collision that creates
# (e.g. with a key like ",") by appending a numeric suffix.
if not ident or not ident[0].isalpha():
ident = "x" + ident
return ident
# ISO C11 language keywords (§6.4.1) and underscore-capital keywords (§7.1.3).
# These may never appear as identifiers.
_C11_KEYWORDS = frozenset({
"auto", "break", "case", "char", "const", "continue", "default", "do",
"double", "else", "enum", "extern", "float", "for", "goto", "if",
"inline", "int", "long", "register", "restrict", "return", "short",
"signed", "sizeof", "static", "struct", "switch", "typedef", "union",
"unsigned", "void", "volatile", "while",
"_Alignas", "_Alignof", "_Atomic", "_Bool", "_Complex", "_Generic",
"_Imaginary", "_Noreturn", "_Static_assert", "_Thread_local",
})
# The keywords GCC and Clang add outside the reserved namespace in their GNU
# modes (-std=gnu*, which is what a plain `cc` compiles with); -std=c11 still
# accepts them as identifiers, so a name that compiles there fails to parse
# for a consumer who does not pass it.
_GNU_KEYWORDS = frozenset({
"asm", "typeof",
})
# <stdbool.h> — Boolean type and values (C11 §7.18).
# bool/true/false are object-like macros; using them as identifiers triggers
# macro expansion and produces a syntax error.
_STDBOOL_NAMES = frozenset({
"bool", "true", "false",
})
# <stddef.h> — Common definitions (C11 §7.19).
# NULL is an object-like macro; offsetof is function-like but still reserved.
_STDDEF_NAMES = frozenset({
"ptrdiff_t", "size_t", "max_align_t", "wchar_t",
"NULL", "offsetof",
})
# <stdint.h> — Integer types with specified widths (C11 §7.20).
_STDINT_NAMES = frozenset({
# Exact-width (optional; universally present on POSIX targets)
"int8_t", "int16_t", "int32_t", "int64_t",
"uint8_t", "uint16_t", "uint32_t", "uint64_t",
# Minimum-width (required)
"int_least8_t", "int_least16_t", "int_least32_t", "int_least64_t",
"uint_least8_t", "uint_least16_t", "uint_least32_t", "uint_least64_t",
# Fastest minimum-width (required)
"int_fast8_t", "int_fast16_t", "int_fast32_t", "int_fast64_t",
"uint_fast8_t", "uint_fast16_t", "uint_fast32_t", "uint_fast64_t",
# Pointer-width (optional; universally present on POSIX targets)
"intptr_t", "uintptr_t",
# Maximum-width (required)
"intmax_t", "uintmax_t",
})
# <stdio.h> — I/O types (C11 §7.21).
# Included in the generated .c file for snprintf.
_STDIO_NAMES = frozenset({
"FILE", "fpos_t",
})
# <stdlib.h> — General-utilities types (C11 §7.22).
# Included in the generated .c file for bsearch/free/malloc.
_STDLIB_NAMES = frozenset({
"div_t", "ldiv_t", "lldiv_t",
})
# <string.h> — String types (C11 §7.24).
# Included in the generated .c file for memcmp; introduces no new type names
# beyond size_t (already in _STDDEF_NAMES).
# POSIX.1-2008 extensions used by the project's codec/json.h API.
_POSIX_NAMES = frozenset({
"ssize_t",
})
# The object-like macros ISO C defines in the headers the generated files
# include: the header pulls in <stddef.h>, <stdbool.h>, <stdint.h> (directly
# and through codec/json.h), and the .c adds <string.h>, <stdlib.h>,
# <limits.h>, <stdio.h>, <math.h> and <float.h>. An identifier spelled like
# one of these is macro-expanded wherever it appears -- a struct member, an
# enumerator, a struct tag -- and the generated code no longer parses.
# Function-like macros (INT8_C, isnan, ...) are left out: every generated
# identifier that is followed by '(' carries a lowercase verb, so none can
# spell one.
_STDINT_MACROS = frozenset(
[f"{pfx}INT{w}_MAX" for pfx in ("", "U") for w in (8, 16, 32, 64)]
+ [f"INT{w}_MIN" for w in (8, 16, 32, 64)]
+ [f"{pfx}INT_{kind}{w}_MAX" for pfx in ("", "U")
for kind in ("LEAST", "FAST") for w in (8, 16, 32, 64)]
+ [f"INT_{kind}{w}_MIN" for kind in ("LEAST", "FAST")
for w in (8, 16, 32, 64)]
+ ["INTPTR_MIN", "INTPTR_MAX", "UINTPTR_MAX",
"INTMAX_MIN", "INTMAX_MAX", "UINTMAX_MAX",
"PTRDIFF_MIN", "PTRDIFF_MAX", "SIG_ATOMIC_MIN", "SIG_ATOMIC_MAX",
"SIZE_MAX", "WCHAR_MIN", "WCHAR_MAX", "WINT_MIN", "WINT_MAX"]
)
# <limits.h> — Sizes of integer types (C11 §5.2.4.2.1).
_LIMITS_MACROS = frozenset({
"CHAR_BIT", "SCHAR_MIN", "SCHAR_MAX", "UCHAR_MAX", "CHAR_MIN", "CHAR_MAX",
"MB_LEN_MAX", "SHRT_MIN", "SHRT_MAX", "USHRT_MAX", "INT_MIN", "INT_MAX",
"UINT_MAX", "LONG_MIN", "LONG_MAX", "ULONG_MAX", "LLONG_MIN", "LLONG_MAX",
"ULLONG_MAX",
})
# <stdio.h> (C11 §7.21.1), <stdlib.h> (§7.22) and <math.h> (§7.12).
_STDIO_MACROS = frozenset({
"BUFSIZ", "EOF", "FILENAME_MAX", "FOPEN_MAX", "L_tmpnam",
"SEEK_CUR", "SEEK_END", "SEEK_SET", "TMP_MAX",
"stderr", "stdin", "stdout",
})
_STDLIB_MACROS = frozenset({
"EXIT_FAILURE", "EXIT_SUCCESS", "MB_CUR_MAX", "RAND_MAX",
})
_MATH_MACROS = frozenset({
"HUGE_VAL", "HUGE_VALF", "HUGE_VALL", "INFINITY", "NAN",
"FP_INFINITE", "FP_NAN", "FP_NORMAL", "FP_SUBNORMAL", "FP_ZERO",
"FP_FAST_FMA", "FP_FAST_FMAF", "FP_FAST_FMAL", "FP_ILOGB0", "FP_ILOGBNAN",
"MATH_ERRNO", "MATH_ERREXCEPT", "math_errhandling",
})
# <float.h> — Characteristics of floating types (C11 §5.2.4.2.2); the fast
# backend's .c includes it for DBL_EPSILON. *_NORM_MAX is C2x, but Clang's
# <float.h> defines it in every GNU mode.
_FLOAT_MACROS = frozenset(
[f"{t}_{m}" for t in ("FLT", "DBL", "LDBL") for m in (
"DECIMAL_DIG", "DIG", "EPSILON", "HAS_SUBNORM", "MANT_DIG", "MAX",
"MAX_10_EXP", "MAX_EXP", "MIN", "MIN_10_EXP", "MIN_EXP", "NORM_MAX",
"TRUE_MIN")]
+ ["DECIMAL_DIG", "FLT_EVAL_METHOD", "FLT_RADIX", "FLT_ROUNDS"]
)
# The system-name macros GCC and Clang predefine outside the reserved
# namespace in their default GNU modes (-std=gnu*, which is what a plain `cc`
# compiles with); -std=c11 withholds them, so a name that compiles there
# fails for a consumer who does not pass it.
_GNU_PREDEFINED_MACROS = frozenset({
"i386", "linux", "unix",
})
# The object-like macros the same headers define outside ISO C once a
# feature-test macro is on: POSIX.1-2008 (-D_POSIX_C_SOURCE=200809L, which
# examples/ builds with), X/Open, glibc's _DEFAULT_SOURCE (implied by
# -std=gnu*, the mode the downstream projects compile in) and _GNU_SOURCE
# (which they define on Linux). Past ISO C the set is platform-dependent;
# this is what glibc exposes to GCC and Clang under -std=gnu11 -D_GNU_SOURCE,
# a superset of every narrower mode, and scripts/gen_schema_test.py checks it
# against the compiler in each of them.
#
# The integer width macros (ISO/IEC TS 18661-1, later C2x) of <stdint.h> and
# <limits.h>.
_WIDTH_MACROS = frozenset(
[f"{pfx}INT{w}_WIDTH" for pfx in ("", "U") for w in (8, 16, 32, 64)]
+ [f"{pfx}INT_{kind}{w}_WIDTH" for pfx in ("", "U")
for kind in ("LEAST", "FAST") for w in (8, 16, 32, 64)]
+ [f"{t}_WIDTH" for t in (
"INTPTR", "UINTPTR", "INTMAX", "UINTMAX", "PTRDIFF", "SIG_ATOMIC",
"SIZE", "WCHAR", "WINT", "CHAR", "SCHAR", "UCHAR", "SHRT", "USHRT",
"INT", "UINT", "LONG", "ULONG", "LLONG", "ULLONG", "BOOL")]
+ ["BOOL_MAX"]
)
# <limits.h>: the POSIX/XSI limits, plus GNU's LONG_LONG_* spellings.
_POSIX_LIMITS_MACROS = frozenset({
"AIO_PRIO_DELTA_MAX", "BC_BASE_MAX", "BC_DIM_MAX", "BC_SCALE_MAX",
"BC_STRING_MAX", "CHARCLASS_NAME_MAX", "COLL_WEIGHTS_MAX",
"DELAYTIMER_MAX", "EXPR_NEST_MAX", "HOST_NAME_MAX", "IOV_MAX",
"LINE_MAX", "LOGIN_NAME_MAX", "LONG_BIT", "MAX_CANON", "MAX_INPUT",
"MQ_PRIO_MAX", "NAME_MAX", "NGROUPS_MAX", "NL_ARGMAX", "NL_LANGMAX",
"NL_MSGMAX", "NL_NMAX", "NL_SETMAX", "NL_TEXTMAX", "NZERO", "PATH_MAX",
"PIPE_BUF", "PTHREAD_DESTRUCTOR_ITERATIONS", "PTHREAD_KEYS_MAX",
"PTHREAD_STACK_MIN", "RE_DUP_MAX", "RTSIG_MAX", "SEM_VALUE_MAX",
"SSIZE_MAX", "TTY_NAME_MAX", "WORD_BIT", "XATTR_LIST_MAX",
"XATTR_NAME_MAX", "XATTR_SIZE_MAX",
"LONG_LONG_MIN", "LONG_LONG_MAX", "ULONG_LONG_MAX",
})
# <stdio.h>: POSIX's L_ctermid/P_tmpdir, and glibc's L_cuserid, SEEK_DATA/HOLE
# and renameat2 flags.
_POSIX_STDIO_MACROS = frozenset({
"L_ctermid", "L_cuserid", "P_tmpdir", "SEEK_DATA", "SEEK_HOLE",
"RENAME_EXCHANGE", "RENAME_NOREPLACE", "RENAME_WHITEOUT",
})
# <stdlib.h>: the <sys/wait.h> flags POSIX has it define, and what glibc's
# <sys/types.h> brings in through it (<endian.h>, <sys/select.h>).
_POSIX_STDLIB_MACROS = frozenset({
"WCONTINUED", "WEXITED", "WNOHANG", "WNOWAIT", "WSTOPPED", "WUNTRACED",
"BIG_ENDIAN", "BYTE_ORDER", "LITTLE_ENDIAN", "PDP_ENDIAN",
"FD_SETSIZE", "NFDBITS",
})
# <math.h>: the XSI constants (M_PI, ...) and their glibc per-type spellings,
# X/Open's MAXFLOAT, and glibc's TS 18661 macros.
_FLOATN = ("F32", "F64", "F128", "F32X", "F64X")
_POSIX_MATH_MACROS = frozenset(
[c + sfx for c in (
"M_E", "M_LOG2E", "M_LOG10E", "M_LN2", "M_LN10", "M_PI", "M_PI_2",
"M_PI_4", "M_1_PI", "M_2_PI", "M_2_SQRTPI", "M_SQRT2", "M_SQRT1_2")
for sfx in ("", "f", "l", *(n.lower() for n in _FLOATN))]
+ ["HUGE_VAL_" + n for n in _FLOATN]
+ ["SNAN", "SNANF", "SNANL", *("SNAN" + n for n in _FLOATN)]
+ ["MAXFLOAT", "FP_LLOGB0", "FP_LLOGBNAN",
"FP_INT_DOWNWARD", "FP_INT_TONEAREST", "FP_INT_TONEARESTFROMZERO",
"FP_INT_TOWARDZERO", "FP_INT_UPWARD"]
)
# The subset of the names above that are only typedef names (see
# _TAG_RESERVED).
_TYPEDEF_NAMES = (
_STDINT_NAMES | _STDIO_NAMES | _STDLIB_NAMES | _POSIX_NAMES
| (_STDDEF_NAMES - {"NULL", "offsetof"})
)
# Combined set used by _make_field_map to detect name collisions.
_C_KEYWORDS = (
_C11_KEYWORDS | _GNU_KEYWORDS | _STDBOOL_NAMES | _STDDEF_NAMES |
_STDINT_NAMES | _STDIO_NAMES | _STDLIB_NAMES | _POSIX_NAMES |
_STDINT_MACROS | _LIMITS_MACROS | _STDIO_MACROS | _STDLIB_MACROS |
_MATH_MACROS | _FLOAT_MACROS | _GNU_PREDEFINED_MACROS |
_WIDTH_MACROS | _POSIX_LIMITS_MACROS | _POSIX_STDIO_MACROS |
_POSIX_STDLIB_MACROS | _POSIX_MATH_MACROS
)
def _escape_keyword(ident: str) -> str:
"""Return *ident* made safe to use as a C identifier.
A name that collides with a C keyword (including GNU mode's ``typeof``), a
reserved type name, or one of the object-like macros the generated code
has in scope in ISO, POSIX or GNU mode (``bool``, ``NULL``, ``SIZE_MAX``,
``EOF``, ``PATH_MAX``, ``M_PI``, GNU's ``unix``, ...) gets a trailing
underscore, which is the same rule ``_make_field_map`` applies to property
names."""
return ident + "_" if ident in _C_KEYWORDS else ident
def _join_ident(*parts: str) -> str:
"""Join identifier segments with single '_' separators, never doubling.
Empty segments are dropped. A separator is inserted between two kept
segments only where neither side already carries one, so a segment ending
in '_' -- a keyword-escaped or otherwise underscore-terminated schema stem
such as ``int_`` (from ``int``) or ``foo_`` (from ``foo-``) -- absorbs the
separator instead of forming an internal ``__`` (see to_c_ident). For
segments that neither end nor begin with '_' -- every scope name, ename and
lookup name a schema without such a stem produces -- the result is
byte-identical to ``"_".join(p for p in parts if p)``, so existing output
is unchanged.
"""
out = ""
for p in parts:
if not p:
continue
if out and not out.endswith("_") and not p.startswith("_"):
out += "_"
out += p
return out
def _count_member(fname: str) -> str:
"""Return the name of an array field's element-count companion member."""
return _join_ident(fname, "count")
def _json_member(fname: str) -> str:
"""Return the member name of a dynamic (raw JSON fragment) field."""
return _join_ident(fname, "json")
def _derived_member(kind: str, fname: str) -> "str | None":
"""Return the companion struct member a field of *kind* also occupies.
``generate_structs_h`` gives an array field a ``<fname>_count`` companion
and names a dynamic field's member ``<fname>_json``; both share the
struct's member namespace with the plain field names, so
``_make_field_map`` has to reserve them. Both are joined through
_join_ident, so a keyword-escaped ``for_`` gets ``for_count``, not
``for__count``. Every other kind occupies its plain name only.
"""
if _is_array_kind(kind):
return _count_member(fname)
if kind == "dynamic":
return _json_member(fname)
return None
def _make_field_map(keys: list, node: dict) -> dict:
"""Return a collision-free mapping from JSON property name → C field name.
Handles four cases in order:
1. Runs of non-identifier characters collapsed to a single '_'; a name not
starting with a letter prefixed with 'x' (both via to_c_ident).
2. C reserved-word conflicts -- a keyword, type name or macro in scope,
codec/json.h's include guard included: append '_' suffix (e.g. 'int'
→ 'int_').
3. Duplicate names after steps 1-2: append '_2', '_3', ... (through
_join_ident, so an escaped 'int_' becomes 'int_2', not 'int__2'),
sorted by (C identifier, JSON key) so the result is deterministic: all
colliding keys are grouped together and the alphabetically-first JSON
key within each collision group keeps the plain name.
4. The companion member names generate_structs_h derives from a field name
(see _derived_member) are reserved alongside it, and a name is
accepted only when its own companion is free too. Without this a
schema pairing ``tags`` (array) with a ``tags_count`` property -- or
``raw`` (dynamic) with ``raw_json`` -- generates a struct with a
duplicate member, which the generator accepts and the C compiler then
rejects with a message naming neither the schema nor the property.
"""
# Steps 1-2: raw conversion for every key. Wherever the struct is
# defined, codec/json.h is included, and its include guard is an empty
# object-like macro like any other in scope.
raw = {}
for key in keys:
ident = to_c_ident(key)
raw[key] = ident + "_" if ident in _JSON_H_MACROS \
else _escape_keyword(ident)
# Step 4 input: the companion member each key's kind brings along. A node
# without usable properties (never produced by collect_scopes, but the map
# builders take an arbitrary node) simply contributes none.
props = node.get("properties", {}) if isinstance(node, dict) else {}
kinds = {}
for key in keys:
prop = props.get(key)
kinds[key] = _infer_c_type(prop)["kind"] \
if isinstance(prop, dict) else None
# Step 3: resolve duplicates. Sort by (C identifier, JSON key) so that
# colliding keys are grouped together and the assignment of plain name vs
# suffix depends only on the C identifier ordering, not on JSON key
# characters that have no C meaning (e.g. '-' vs '_').
used: set = set() if _PRESENCE else {"json_defaults"}
result: dict = {}
for key in sorted(keys, key=lambda k: (raw[k], k)):
base = raw[key]
cand, n = base, 1
while cand in used or _derived_member(kinds[key], cand) in used or (_PRESENCE and "has_" + cand in used):
n += 1
cand = _join_ident(base, str(n))
used.add(cand)
if _PRESENCE:
used.add("has_" + cand)
companion = _derived_member(kinds[key], cand)
if companion is not None:
used.add(companion)
result[key] = cand
return result
def _make_enum_const_map(keys: list, node: dict, ename: str) -> dict:
"""Return a collision-free mapping JSON key → full enum-constant name.
An enum constant is ``<ENAME>_<FIELD>`` with both parts uppercased. The
C field names are unique only case-sensitively (``_make_field_map``), so
uppercasing can collapse two of them onto the same token (e.g. keys ``id``
and ``ID`` → field names ``id`` and ``ID`` → both ``ID``), which would emit
a duplicate enumerator and duplicate ``switch`` case label. Disambiguate
the same way ``_make_field_map`` disambiguates field names: append
``_2``, ``_3``, ... (through ``_join_ident``) in a deterministic order.
Keys with no case collision keep the plain ``<ENAME>_<FIELD>`` name, so
existing output is unchanged.
"""
fname_map = _make_field_map(keys, node)
base = {key: fname_map[key].upper() for key in keys}
used: set = set()
result: dict = {}
for key in sorted(keys, key=lambda k: (base[k], fname_map[k], k)):
cand = base[key]
if cand in used:
n = 2
while _join_ident(cand, str(n)) in used:
n += 1
cand = _join_ident(cand, str(n))
used.add(cand)
result[key] = _join_ident(ename.upper(), cand)
return result
# ---------------------------------------------------------------------------
# Schema traversal
# ---------------------------------------------------------------------------
def _canonicalize(node):
"""Return a canonical copy of a JSON Schema node.
Equivalent schemas (same content, any property order) are collapsed to a
unique representation so all downstream code always sees the same input.
Normalizations applied recursively:
* ``properties`` dict — keys sorted alphabetically.
* ``$defs`` / ``definitions`` dict — keys sorted alphabetically.
* ``required`` array — sorted and deduplicated.
* Subschemas — recursively canonicalized; instance data stays verbatim.
"""
if isinstance(node, list):
return [_canonicalize(item) for item in node]
if not isinstance(node, dict):
return node
result = {}
for k, v in node.items():
if k in ("properties", "$defs", "definitions", "dependentSchemas") \
and isinstance(v, dict):
result[k] = {name: _canonicalize(child)
for name, child in sorted(v.items())}
elif k == "required" and isinstance(v, list):
result[k] = sorted(set(v))
elif k in ("items", "allOf", "anyOf", "oneOf", "not", "if", "then",
"else", "prefixItems", "contains", "additionalProperties",
"propertyNames", "unevaluatedItems", "unevaluatedProperties"):
result[k] = _canonicalize(v)
else:
result[k] = v
return result
# Keywords whose value JSON Schema requires to be a number, and those whose
# value it requires to be a non-negative integer (an integer-valued number such
# as 2.0 included).
_NUMBER_KEYWORDS = frozenset({
"minimum", "maximum", "exclusiveMinimum", "exclusiveMaximum",
"multipleOf",
})
_COUNT_KEYWORDS = frozenset({
"minLength", "maxLength", "minItems", "maxItems", "minProperties",
"maxProperties", "minContains", "maxContains",
})
def _keyword_value_error(key: str, value) -> "str | None":
"""Explain why *value* is not a legal value of the numeric keyword *key*,
or return None when it is.
A boolean is refused explicitly: bool is a Python subclass of int, so it
would otherwise pass as the number 0 or 1 -- and be spelled ``True`` in the
emitted C. A non-finite float is refused too: Python's json module accepts
NaN and Infinity (and an overflowing literal such as 1e400 decodes to
Infinity), but none of them is a JSON number.
"""
if isinstance(value, bool) or not isinstance(value, (int, float)):
shown = "" if value is None else f" {json.dumps(value)}"
return f"must be a number, not the JSON {_json_type_name(value)}{shown}"
if isinstance(value, float) and not math.isfinite(value):
return f"must be a finite number, not {value!r}"
if key in _COUNT_KEYWORDS and (value < 0 or value != math.floor(value)):
return f"must be a non-negative integer, not {value!r}"
return None
def _reject_malformed_keywords(node, path="") -> None:
"""Reject, at schema-processing time, a keyword value the JSON Schema
meta-schema forbids and no generated check could honor: an empty
``enum``, a malformed ``required``, and a numeric keyword
(_NUMBER_KEYWORDS, _COUNT_KEYWORDS) whose value is not a number of the
kind the keyword takes.
Every schema node is screened, whether or not the generator enforces the
keyword there, so both backends refuse exactly the same schemas before
either reads a bound."""
if isinstance(node, dict):
enum = node.get("enum")
if isinstance(enum, list) and not enum:
sys.exit(
f"error: {path or '<root>'}: enum must contain at least "
"one item (an empty enum is unsatisfiable)")
if "required" in node:
required = node["required"]
if not isinstance(required, list):
sys.exit(
f"error: {path or '<root>'}: required must be an array "
f"of property-name strings, not the JSON "
f"{_json_type_name(required)}")
for name in required:
if not isinstance(name, str):
sys.exit(
f"error: {path or '<root>'}: required must contain "
f"only property-name strings, not the JSON "
f"{_json_type_name(name)}")
for key in sorted(_NUMBER_KEYWORDS | _COUNT_KEYWORDS):
if key not in node:
continue
why = _keyword_value_error(key, node[key])
if why is not None:
sys.exit(f"error: {path or '<root>'}: {key} {why}")
for key, value in node.items():
if key == "properties" and isinstance(value, dict):
for prop, child in value.items():
child_path = f"{path}.{prop}" if path else prop
_reject_malformed_keywords(child, child_path)
elif key == "items":
_reject_malformed_keywords(value, path + "[]")
elif key in ("$defs", "definitions", "dependentSchemas") \
and isinstance(value, dict):
# a name -> subschema map: its keys are names, not keywords
for name, child in value.items():
child_path = f"{path}.{name}" if path else name
_reject_malformed_keywords(child, child_path)
elif key in ("allOf", "anyOf", "oneOf", "not", "if", "then",
"else", "prefixItems", "contains",
"additionalProperties", "propertyNames",
"unevaluatedItems", "unevaluatedProperties"):
_reject_malformed_keywords(value, path)
elif isinstance(node, list):
for value in node:
_reject_malformed_keywords(value, path)
def collect_scopes(node, prefix: str, path: str = "",
paths: "dict | None" = None):
"""Plan fixed objects and maps; map container values remain opaque."""
if not isinstance(node, dict):
return
if node.get("type") == "object" and _infer_c_type(node)["kind"] != "dynamic":
name = prefix if not path else _join_ident(prefix, to_c_ident(path))
keys = sorted(node.get("properties", {}))
if paths is not None:
paths[name] = path
yield name, keys, node
for prop, child in sorted(node.get("properties", {}).items()):
child_path = f"{path}.{prop}" if path else prop
yield from collect_scopes(child, prefix, child_path, paths)
elif node.get("type") == "array" and _infer_c_type(node)["kind"] == "array_object":
yield from collect_scopes(node["items"], prefix, path, paths)
def _required_set(node: dict) -> set:
"""Return the set of required property names from a JSON Schema object node."""
return set(node.get("required", []))
def _is_dynamic_object(prop_schema: dict) -> bool:
"""True when the schema node is an object with no fixed ``properties``
(missing, or an empty ``properties: {}``), meaning it carries no named
fields and its content is kept as a raw JSON fragment at runtime.
An empty ``properties: {}`` must be treated as dynamic too: collect_scopes
only emits a struct scope when a property set is non-empty, so classifying
an empty-property object as a struct field would reference a ``struct`` type
that is never defined (and helper functions never generated)."""
return (
isinstance(prop_schema, dict)
and prop_schema.get("type") == "object"
and not prop_schema.get("properties")
)
def _infer_c_type(prop_schema: dict) -> dict:
"""Return a descriptor dict for a single JSON Schema property.
Keys:
kind : 'string' | 'int' | 'uint' | 'double' | 'bool' |
'object' | 'dynamic' | 'array_string' |
'array_object' | 'array_primitive'
c_type : C type for the struct field: 'int' or 'intmax_t' (int
kind), 'unsigned' or 'uintmax_t' (uint kind)
c_base : the C base type string (for primitive arrays)
A boolean subschema (``true`` admits anything, ``false`` nothing) carries
no type, and neither does tuple-form ``items`` (a list of subschemas): both
are stored as a raw JSON fragment, and _dynamic_reason says why.
"""
if not isinstance(prop_schema, dict):
return {"kind": "dynamic"}
t = prop_schema.get("type")
if t == "string":
return {"kind": "string"}
if t == "integer":
# JSON Schema allows non-integral bounds on integer types; tighten
# them to the nearest integer in the valid direction.
minimum = prop_schema.get("minimum")
maximum = prop_schema.get("maximum")
if minimum is not None:
minimum = math.ceil(minimum)
if maximum is not None:
maximum = math.floor(maximum)
# No single 64-bit C integer type spans the range → store as a raw
# JSON fragment. Besides min < INT64_MIN or max > UINT64_MAX, a
# negative minimum with a maximum above INT64_MAX fits neither
# intmax_t (max too large) nor uintmax_t (minimum negative).
if (minimum is not None and minimum < _INT64_MIN) or \
(maximum is not None and maximum > _UINT64_MAX) or \
(minimum is not None and minimum < 0 and
maximum is not None and maximum > _INT64_MAX):
return {"kind": "dynamic"}
# Both bounds explicit → pick the narrowest 32-bit type if possible.
if minimum is not None and maximum is not None:
if minimum >= 0 and maximum <= _UINT32_MAX:
return {"kind": "uint", "c_type": "unsigned"}
if minimum >= _INT32_MIN and maximum <= _INT32_MAX:
return {"kind": "int", "c_type": "int"}
# One or no bound → widest appropriate type.
if minimum is not None and minimum >= 0:
return {"kind": "uint", "c_type": "uintmax_t"}
return {"kind": "int", "c_type": "intmax_t"}
if t == "number":
return {"kind": "double"}
if t == "boolean":
return {"kind": "bool"}
if t == "null":
return {"kind": "null"}
if t == "object":
if _is_dynamic_object(prop_schema):
if "$ref" in prop_schema or "patternProperties" in prop_schema:
return {"kind": "dynamic"}
return {"kind": "map"}
return {"kind": "object"}
if t == "array":
items = prop_schema.get("items", {})
if not isinstance(items, dict):
return {"kind": "dynamic"}
item_t = items.get("type")
if item_t == "string":
return {"kind": "array_string"}
if item_t == "object":
# Unsupported object schemas have no generated child scope.
if _infer_c_type(items)["kind"] == "dynamic":
return {"kind": "dynamic"}
return {"kind": "array_object"}
if item_t == "integer":
elem = _infer_c_type(items)
if elem["kind"] == "dynamic":
return {"kind": "dynamic"}
return {"kind": "array_primitive", "c_base": elem.get("c_type", "intmax_t")}
if item_t == "boolean":
return {"kind": "array_primitive", "c_base": "bool"}
if item_t == "number":
return {"kind": "array_primitive", "c_base": "double"}
return {"kind": "dynamic"}
return {"kind": "dynamic"}
# ---------------------------------------------------------------------------
# gperf generation
# ---------------------------------------------------------------------------
_GPERF_MACROS = (
"TOTAL_KEYWORDS",
"MIN_WORD_LENGTH",
"MAX_WORD_LENGTH",
"MIN_HASH_VALUE",
"MAX_HASH_VALUE",
)
_LINE_DIRECTIVE = re.compile(r'^#line\b')
def _lookup_root(pfx: str, scope_name: str) -> str:
"""Return the shared prefix for every symbol in a scope's lookup table.
All gperf-derived and gperf-adjacent symbols (the public ``lookup``
function, the ``hash`` function, the internal ``kv_lookup_`` helper,
the ``stringpool`` storage macro, and the ``kv`` struct tag) share
this prefix so the naming convention ``<pfx>_lookup_<scope>`` is
visible at a glance -- unless a nested scope's own lookup already takes a
helper's name, in which case _plan_identifiers moves the helpers to
``<pfx>_lookup_<scope>_2`` (see _IdentPlan).
``_join_ident`` normalises the join: empty parts are dropped (so the
default empty prefix yields ``lookup_<scope>``, not a reserved leading
``_``), and a part already ending in ``_`` -- a user prefix like ``ex_``
or a keyword-escaped scope name like ``int_`` -- absorbs the separator
instead of doubling it into an internal ``__`` (see to_c_ident).
"""
pfx = pfx.rstrip("_")
return _join_ident(pfx, "lookup", scope_name)
def _make_gperf_input(lookup_name: str, helper_root: str, keys: list) -> str:
"""Build the complete gperf input for a scope's keys.
gperf's own symbols (the hash and ``kv_lookup_`` functions, the string
pool and the ``kv`` struct tag) are named from *helper_root*, which is
*lookup_name* unless _plan_identifiers had to move them off a name a
nested scope already takes.
Every option is embedded as a ``%`` declaration so the input file is the
single source of truth; gperf is then invoked with no command-line
options (see _run_gperf). Each declaration is the documented equivalent
of a former command-line flag:
%compare-lengths (-l) binary comparison: the generated lookup
compares by length + memcmp instead of
assuming NUL-terminated input, and keys may
contain NUL bytes. Mandatory here.
%enum (-E) emit constants as a local enum, not #define.
%pic (-P) string-pool layout for shared libraries.
%null-strings NULL (not "") for empty table slots.
%define string-pool-name (-Q) name the %pic string pool.
%define initializer-suffix (-F) zero the idx field of empty slots.
(%compare-strncmp / -c is intentionally omitted: gperf documents it as
ignored whenever %compare-lengths is in effect, which it always is here.)
Keys are emitted as double-quoted C-string literals with backslash
escapes -- gperf's quoted-keyword syntax -- so nearly any byte is
representable: delimiters, quotes, '%'/'#', control characters, non-ASCII,
and (because of %compare-lengths) NUL. '?' is the exception, because
gperf owns both ends of that round trip and handles neither: it rejects
the '\\?' escape _c_string_literal emits ("invalid escape sequence in
string"), and it re-emits a decoded '?' verbatim into its output, where a
"??" pair would become a trigraph again. _gperf_safe_keys screens those
keys out -- along with the empty keyword gperf also refuses -- before we
get here.
"""
header_lines = [
"%language=ANSI-C",
"%struct-type",
"%readonly-tables",
"%compare-lengths",
"%enum",
"%pic",
"%null-strings",
f"%define string-pool-name {_join_ident(helper_root, 'stringpool')}",
"%define initializer-suffix ,0",
f"%define hash-function-name {_join_ident(helper_root, 'hash')}",
f"%define lookup-function-name {_join_ident(helper_root, 'kv_lookup_')}",
"%define slot-name name",
"%{",
f"/* {lookup_name}: {len(keys)} keys (auto-generated -- do not edit) */",
"%}",
"struct " + _join_ident(helper_root, "kv") + " { int name; int idx; };",
"%%",
]
keyword_lines = [
f'"{_c_string_literal(key)[0]}", {i}' for i, key in enumerate(keys)
]
return "\n".join(header_lines) + "\n" + "\n".join(keyword_lines) + "\n%%\n"
def _gperf_safe_keys(keys: list) -> bool:
"""True when gperf can build a lookup table for these keys.
Keys are emitted as quoted C-string literals (see _make_gperf_input), so
nearly every byte is representable -- delimiters, quotes, control
characters, non-ASCII, and (under %compare-lengths) NUL. Two cases must
fall back to the bsearch lookup, which emits its keys through
_c_string_literal directly and has neither restriction:
* the empty key, the one keyword gperf refuses outright ("Empty input
keyword is not allowed");
* a key containing '?', which gperf can neither read (it rejects the
'\\?' escape _c_string_literal emits) nor write (it prints a decoded
'?' unescaped, re-forming any trigraph in its own output).
"""
return all(len(key) > 0 and "?" not in key for key in keys)
def _run_gperf(lookup_name: str, helper_root: str, keys: list) -> str:
# All options are embedded as %-declarations in the input (see
# _make_gperf_input), so gperf is invoked with no argv options and the
# input is fed on stdin.
src = _make_gperf_input(lookup_name, helper_root, keys)
try:
r = subprocess.run(
["gperf"],
input=src,
capture_output=True,
text=True,
check=True,
)
return r.stdout
except FileNotFoundError:
sys.exit("error: gperf not found -- install it with: apt install gperf")
except subprocess.CalledProcessError as e:
sys.exit("error: gperf failed for scope " +
repr(lookup_name) + ":\n" + e.stderr)
def _suppress_unused_hash_params(lines: list, hash_name: str) -> list:
"""Cast-to-void whichever of the gperf hash's ``str``/``len`` parameters its
generated body does not reference, so the hash never trips
``-Wunused-parameter``.
gperf drops ``len`` from the hash when every key that reaches it has the
same length (the length term is then constant), and drops ``str`` for a
pure length hash, so which parameter is unused depends on the key set and
must be detected rather than assumed. Casting an *used* parameter to void
as well would be harmless, but emitting only what is needed keeps the
generated output minimal.
"""
sig = hash_name + " (register const char *str, register size_t len)"
sig_idx = next(
(i for i, ln in enumerate(lines) if ln.rstrip().endswith(sig)), None)
if sig_idx is None:
return lines
brace_idx = next(
(i for i in range(sig_idx + 1, len(lines)) if lines[i].strip() == "{"),
None)
if brace_idx is None:
return lines
# Find the function's closing brace by tracking brace depth from the opener
# (the asso_values initializer braces balance out along the way).