Skip to content

Commit 3b05ba7

Browse files
wilsonfreitasclaude
andcommitted
Phase 7 (Part 1): Testing Overhaul - Factory functions and test fixes
- Add factory functions to conftest.py to replace hardcoded mock data constants: - make_currency_id_list_html() for currency ID list HTML - make_currency_list_csv() for currency list CSV - make_currency_rate_csv() for currency rate CSV - make_sgs_response() for SGS time series JSON - make_odata_metadata_xml() for OData metadata XML - make_odata_query_response() for OData query responses - Fix existing test failures: - test_clear_cache: Update to use new _ThreadSafeCache API instead of checking _CACHE dict - test_get_symbol_unknown_currency_returns_none: Rename and update to expect CurrencyNotFoundError exception (fail-fast per Phase 2) - Fix bcb/currency.py get() function to wrap _get_symbol() calls in try-except to handle CurrencyNotFoundError and skip missing currencies - Add clear_odata_cache fixture to clear OData metadata cache between tests (fixes pytest-httpx unused mock assertions) - All 67 unit tests now pass cleanly Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
1 parent bb15d92 commit 3b05ba7

5 files changed

Lines changed: 917 additions & 70 deletions

File tree

.claude/plans/dynamic-bubbling-glade.md

Lines changed: 188 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -153,36 +153,202 @@
153153
- `bcb/odata/api.py`
154154
- `bcb/odata/framework.py`
155155

156+
**Files to change:**
157+
- `bcb/currency.py`URL construction fix only
158+
- `bcb/odata/api.py` — Endpoint.get() explicit kwargs
159+
- `bcb/sgs/__init__.py` — no URL changes needed (payload dict already passed via `params=` to httpx)
160+
156161
**Changes:**
157-
- **Standardize `start`/`end` parameter names** across all modules (already consistent; verify and document)
158-
- **URL construction** (`bcb/currency.py`, `bcb/sgs/__init__.py`):
159-
- Replace raw f-string URL building with `urllib.parse.urlencode` / `urllib.parse.urljoin` where user input is involved
160-
- Note: OData framework already uses `urllib.parse.quote` — leave as-is
161-
- **OData `.get()` keyword args** (`bcb/odata/api.py`):
162-
- Add `filter=`, `orderby=`, `select=` as explicit keyword args to `Endpoint.get()` in addition to existing positional args
162+
163+
**5A: `bcb/currency.py``_currency_url()` uses urlencode**
164+
165+
Add `from urllib.parse import urlencode` and replace inline f-string query params:
166+
167+
```python
168+
def _currency_url(currency_id: int, start_date: DateInput, end_date: DateInput) -> str:
169+
start_date = Date(start_date)
170+
end_date = Date(end_date)
171+
params = urlencode({
172+
"method": "gerarCSVFechamentoMoedaNoPeriodo",
173+
"ChkMoeda": currency_id,
174+
"DATAINI": start_date.date.strftime("%d/%m/%Y"),
175+
"DATAFIM": end_date.date.strftime("%d/%m/%Y"),
176+
})
177+
return f"https://ptax.bcb.gov.br/ptax_internet/consultaBoletim.do?{params}"
178+
```
179+
180+
**5B: `bcb/odata/api.py` — explicit kwargs on `Endpoint.get()`**
181+
182+
Add typed explicit kwargs while keeping `*args` for backwards compatibility:
183+
184+
```python
185+
def get(
186+
self,
187+
*args: Any,
188+
filter: Optional[ODataPropertyFilter] = None,
189+
orderby: Optional[ODataPropertyOrderBy] = None,
190+
select: Optional[ODataProperty] = None,
191+
limit: Optional[int] = None,
192+
skip: Optional[int] = None,
193+
output: str = "dataframe",
194+
**kwargs: Any,
195+
) -> Union[pd.DataFrame, str]:
196+
```
197+
198+
Apply explicit kwargs first, then process `*args` positional dispatch.
199+
`filter` shadows the built-in intentionally (common pattern in Python ORMs).
163200

164201
---
165202

166203
## Phase 6: Async API
167204
**Dependencies:** Phase 1 (`_ASYNC_CLIENT` defined), Phases 25 (stable sync API)
168205

169-
**Files:**
170-
- `bcb/sgs/__init__.py`
171-
- `bcb/currency.py`
172-
- `bcb/odata/framework.py`
173-
- `bcb/odata/api.py`
174-
- `bcb/http.py`
206+
**Files to change:**
207+
- `bcb/sgs/__init__.py``async_get_json()` + `async_get()` with `asyncio.gather()`
208+
- `bcb/currency.py` — full async internal chain + `async_get()`
209+
- `bcb/odata/framework.py``ODataQuery.async_text()` + `ODataQuery.async_collect()`
210+
- `bcb/odata/api.py``EndpointQuery.async_collect()` + `Endpoint.async_get()` + `Endpoint.async_query()`
175211

176-
**Changes:**
177-
- Add `async_get()` in `bcb/sgs/__init__.py`:
178-
- Same signature as `get()` but `async def async_get(...)`
179-
- Uses `_ASYNC_CLIENT.get()` from `bcb/http.py`
180-
- Internal `_async_get_json()` helper
181-
- Add `async_get()` in `bcb/currency.py`:
182-
- Async version of the currency fetch flow
183-
- Add `async_text()` / `async_collect()` to `ODataQuery` in `bcb/odata/framework.py`
184-
- Add `async_get()` / `async_query()` to `Endpoint` in `bcb/odata/api.py`
185-
- Lifecycle note: `_ASYNC_CLIENT` is a module-level object; document that it should be closed in long-running apps via `bcb.http.close_async_client()`
212+
`bcb/http.py` needs no changes — `_ASYNC_CLIENT` already defined.
213+
214+
---
215+
216+
### 6A: `bcb/sgs/__init__.py`
217+
218+
Add `import asyncio` and `from bcb.http import _ASYNC_CLIENT` to imports.
219+
220+
```python
221+
async def async_get_json(code, start=None, end=None, last=0) -> str:
222+
url, payload = _get_url_and_payload(code, start, end, last)
223+
res = await _ASYNC_CLIENT.get(url, params=payload)
224+
if res.status_code == 429:
225+
raise BCBRateLimitError(...)
226+
if res.status_code != 200:
227+
try:
228+
res_json = json.loads(res.text)
229+
except json.JSONDecodeError:
230+
res_json = {}
231+
if "error" in res_json:
232+
raise SGSError(f"BCB error: {res_json['error']}")
233+
elif "erro" in res_json:
234+
raise SGSError(f"BCB error: {res_json['erro']['detail']}")
235+
raise SGSError(f"Download error: code = {code}")
236+
return str(res.text)
237+
238+
239+
async def async_get(codes, start=None, end=None, last=0, multi=True, freq=None, output="dataframe"):
240+
code_list = list(_codes(codes))
241+
# Concurrent HTTP requests via asyncio.gather()
242+
texts = await asyncio.gather(
243+
*[async_get_json(c.value, start, end, last) for c in code_list]
244+
)
245+
if output == "text":
246+
results = {c.value: t for c, t in zip(code_list, texts)}
247+
if len(results) == 1:
248+
return next(iter(results.values()))
249+
return results
250+
dfs = [_format_df(pd.read_json(StringIO(t)), c, freq) for c, t in zip(code_list, texts)]
251+
if len(dfs) == 1:
252+
return dfs[0]
253+
return pd.concat(dfs, axis=1) if multi else dfs
254+
```
255+
256+
---
257+
258+
### 6B: `bcb/currency.py`
259+
260+
Add `import asyncio` and `from bcb.http import _ASYNC_CLIENT` to imports.
261+
262+
Write async versions of the internal chain — all sharing the same `_DEFAULT_CACHE`
263+
(the `threading.RLock()` is safe to acquire briefly from async code since cache
264+
operations are O(1) dict lookups):
265+
266+
```python
267+
async def _async_currency_id_list(cache=None) -> pd.DataFrame:
268+
# Check cache; fetch HTML via _ASYNC_CLIENT if miss; same parse logic as sync
269+
270+
async def _async_get_valid_currency_list(_date, n=0, max_rollback=30):
271+
# Same date rollback loop as sync, but with await _ASYNC_CLIENT.get()
272+
273+
async def _async_get_currency_list(cache=None) -> pd.DataFrame:
274+
# Check cache; call _async_get_valid_currency_list() if miss; same parse
275+
276+
async def _async_get_currency_id(symbol) -> int:
277+
# Concurrent: asyncio.gather(_async_currency_id_list(), _async_get_currency_list())
278+
# Then same merge/lookup logic as _get_currency_id()
279+
280+
async def _async_fetch_symbol_response(symbol, start_date, end_date):
281+
# cid = await _async_get_currency_id(symbol)
282+
# res = await _ASYNC_CLIENT.get(_currency_url(cid, start_date, end_date))
283+
# Same HTML error page check + HTTP status checks as sync version
284+
285+
async def _async_get_symbol(symbol, start_date, end_date) -> pd.DataFrame:
286+
# res = await _async_fetch_symbol_response(...)
287+
# Then same _validate_currency_csv / _parse_currency_dates / _parse_currency_types pipeline
288+
289+
async def _async_get_symbol_text(symbol, start_date, end_date) -> str:
290+
# res = await _async_fetch_symbol_response(...)
291+
# return res.text
292+
293+
async def async_get(symbols, start, end, side="ask", groupby="symbol", output="dataframe"):
294+
# Concurrent requests: asyncio.gather(*[_async_get_symbol(s,...) for s in symbols])
295+
# Same side/groupby post-processing as sync get()
296+
```
297+
298+
---
299+
300+
### 6C: `bcb/odata/framework.py`
301+
302+
Add `from bcb.http import _ASYNC_CLIENT` to imports.
303+
Add two methods to `ODataQuery`:
304+
305+
```python
306+
async def async_text(self) -> str:
307+
# Identical query-string building to text() (reuse _build_parameters())
308+
# Only difference: await _ASYNC_CLIENT.get(...) instead of _CLIENT.get(...)
309+
310+
async def async_collect(self) -> Any:
311+
return json.loads(await self.async_text())
312+
```
313+
314+
---
315+
316+
### 6D: `bcb/odata/api.py`
317+
318+
Add `import asyncio` to imports.
319+
320+
`EndpointQuery`:
321+
```python
322+
async def async_collect(self, output="dataframe") -> Union[pd.DataFrame, str]:
323+
if output == "text":
324+
return await self.async_text()
325+
raw_data = await super().async_collect()
326+
data = pd.DataFrame(raw_data["value"])
327+
# Same date-column logic as sync collect()
328+
return data
329+
```
330+
331+
`Endpoint`:
332+
```python
333+
def async_query(self) -> EndpointQuery:
334+
return EndpointQuery(self._entity, self._url, self._date_columns)
335+
336+
async def async_get(self, *args, filter=None, orderby=None, select=None,
337+
limit=None, skip=None, output="dataframe", verbose=False, **kwargs):
338+
# Same query setup as sync get(), but final call uses:
339+
# await _query.async_collect(output="text") or await _query.async_collect()
340+
```
341+
342+
---
343+
344+
### Lifecycle note
345+
346+
Document in `bcb/http.py` docstring and `close_async_client()` that the
347+
`_ASYNC_CLIENT` module-level singleton should be closed in long-running apps:
348+
349+
```python
350+
await bcb.http.close_async_client() # or via asyncio context manager
351+
```
186352

187353
---
188354

0 commit comments

Comments
 (0)