From 0b31ebf801312cefcc4e9828b925a7432be1c01f Mon Sep 17 00:00:00 2001 From: "developtheweb@protonmail.com" Date: Sat, 15 Aug 2026 04:35:09 -0400 Subject: [PATCH] fix(nightly): repoint feed to rss.xml, parse read-time, repair scraper MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The FEED table had been frozen since 2026-07-08 — 37 nightly runs that all went green while updating nothing. Both paths were dead at once, which is why it left no trace: FEED_URL requested /feed.xml, which 404s (the site serves /rss.xml), and the HTML fallback then matched zero articles because it looks for a thumbnail alt= attribute and a Title-Case date, and the redesign has neither — no feature images on the index, and uppercase abbreviated months. It looked current only because no post has been published since July 2025, so the four titles it froze with are still the four newest. FEED_URL now points at the real feed, which also carries read-time in the site's own namespace — so the meta column comes from the publisher instead of being scraped back out of a page. The fallback is rewritten against the markup the index actually emits: one `row` div holding an anchor and a `meta` span. Failures stay fail-safe and stop being invisible. A skipped section now emits a GitHub Actions ::warning annotation that shows up in the run summary, plus a count of how many sections did not update. A silent skip and a successful run looked identical, and that is the whole reason this went unnoticed. Verified against the live site: the fallback parses all 25 articles with dates and read-times, and the feed path yields the same four rows the table already shows. --- engine/__pycache__/nightly.cpython-314.pyc | Bin 0 -> 15133 bytes engine/nightly.py | 118 +++++++++++++++------ 2 files changed, 86 insertions(+), 32 deletions(-) create mode 100644 engine/__pycache__/nightly.cpython-314.pyc diff --git a/engine/__pycache__/nightly.cpython-314.pyc b/engine/__pycache__/nightly.cpython-314.pyc new file mode 100644 index 0000000000000000000000000000000000000000..53a6da00cbab429691ad4a93ca660ce1512de1b2 GIT binary patch literal 15133 zcmbt*Yg8Lowq{AH(pyNpjg3u#4YrJV7-PrA7zYD>8nYi5baX-I`N>6Pg}_aCvnCVkWM zW4?2w5`vKArl)L9JDi)M=-H}T>Di{*>Dh7888cQeCtYe@1#^Y# z*`{~6dhV%Qz1FY(ze{e{TUtF|DQL#Kkz@T?E&Q7Pyy^0}>6Jpk*lbKcQH#G*J(IvT zn=&;^1WqlzQtD&`bCXfsM6WEkDif^g=6XgbK5tOB2ql88lnEKsa=~88s9OaG(rtnh zX@%fI$_sf&D+M>wD#3$vyO57`hfsjDS|~(Xb7iMcbntP83E=COGgoRVndZz3Rp1M6 zr+?nkSP5RLJ2*U|hQ@hS3Q41qDv$HBG$c#P2tO2&d38kMqjF>@7?OCQtG)AFSH0cd z7Y&H2q|nS!QNAw8bwg6vF9mo-@~go}Sm7f>dhyXfi=F3>cXf60u`a@`VmvG?%9~|PNk!UQ0t^#{_DWpiGL$sB1D3W{wy~|QK zAj#-|FccZ)PxhWW>qWT>z3sghNMX`v&g(_S__3fm!iR!4BwkTPxj~kq5nhd`Vo0H5 zZSU<9FkVuR1GhG)>w~^5eNkb@seZ&-EmvybB_iz4wIltaeF`-jfCsJr$oZ>%_FKBRazPv6jiz* zg-3%SF)S%ky+1MHR%Qh#$a6z3Yq2Z`_c`oaxirD0SI$I?UcXEqsKt3*7Jt@Gd^ z6;J({16q2psGxq3WFk6pd1e4F8 zZ}PTzP+_XyrO!RCwfE!>n?56Uq9&gy+hZN=k?C=7P=nKJWJ{NP!}!dC#b5bJ;m#D@a8tb zE;#fyW2Tr}ZDoQ6uPbiv!w#=& z$F@=9WdW?TP$Y^KJSGj+h0}7YVQ+n-*PvNOBoQJ&(ai0BzZ6w9=ZRCjC;N^K^qxP{ z)%|6Oz5>N)uznbuKQ>5~s^)5ssUs0N7^izOZaF52vLy4QK{YrkMPh24i>X6(2Q>?1 zL`d{Y$TCZ^9Fdi7uR}A(ySSRNlC;oEr@N=R z6KyHy?uo8XEzX(3m$$vREy*8T%s=#D{vo}_GjluVtLICTyAIv6wk+G+)7z%DE!rv; zY!$PnIU#AQShO|Xvo+3lF1ZV*$EU`>|J0(pa=~3WyK~-~bXTU_tzVcJ`(Bj7|CFuq zOT~#Xztd9LQNz7kG8iA&L2;fA5M0P_eus+4u6vMKR_YI zpTQR7di9O}J8mW*{Coyr4MUSDb7gEY$sHFP525Q}yhAy2@lkRe@Q~|%G9Y>o;fH!` zlSzN;5rdEE$?Yjq(w8l9mD#g7^AM*f_ci6SV)3eBO?k7Ab23Dr37pSJQx^ATd<6?; z+#KgxE9>g`xV;4`UFhZO>e}Mgmagv3^wnrkQG(%NK5pgHXhhKrcCS&h`@~R8>LQyJ zDhyvs^;$Gmm2Rq*pv_&qT>uu<0dM&w@cOho+6zdLP>C9R?oK?od7-KoazhK%(o$3k) z%5>uvv+L?Rx5$1?i&=I2om=DtbJVCe+&yC-|4KLLc?r%{ZvM`1Wbs*gO7*r@&-z!{nRyhzx&ffFpf^X8@ti?0MXjsgXP70q zU4}D|Ha=FblYP1cPYpd~tF7x-g3)IZoIX}?`OHF|&)Sgv))l0+M;OM&Y(rfFEBXuc zT4p9d`y;U8?2ADiP`Km2?vaDo`@q(bnCu6drg%80N`UB4zaao?DF6W8OQ1O}3AUGm zFiD7Rt|OR$52mdEC0!{p7Usj^sH8*z4)*A%9)^?%t05E{4cmb;P=82{jMC1cvB6MK z83D?GAwW(5?O}dUA`%QJSd0=gM)Y6j0X~QsB?96POZ9e*MTet+06{e*X^h5(g5m3$ z15eKMtEe(6QLUv}iTyUvC!D3DkTvU2FdU$g6tWY>gk~jekm=E^ zcn~F5at&$Rb*{a;??Q)gs;76L`+{Z(DuE#I32;pmm?z-10ro^9s>YGGQ{=s|>dc$yTJjXVYPO?k+#V>Dtacg2MmA`W_zb=_yH`lc2 zXDN+YLlMYxq?M^(}KGxymC# zJFj^m50d`CqzT!R?26H3%U>?svRJrdp>W4yVckMu-JE}Z``?Yc6@D|Es&BiSmnu9z z*|}n7Fcx$+YkcF#Y(sKydve>cl&53Sb1LaMbyr?=_bj-3Qf>j=R(o&vzutfQ>QArE z?3gK;83$^*Q!vSX>hL_j|K;A7dY7E}(`Tp7CU!z)RL>X9-?*dR^?lHrEE5);7d|gz z@~c)hGtT^pvwwK#VmzB4G6q}z$4<|R3Fj{rva8-Xw7;v#_T>{>E=c62Rn^mzwYjo=FqHr^IVCo*#j-+? zSS}mbWxgy8M3vMh6sm9=*<)9;>dd!sepZQnO@0;{$Q5r**>!3&_aadML${b z0jd(u^f5VNttTteb7v)IPGB>xKQL=XKst+Vf$862BVnt5iKuN=l>W(mvOeqjKBM+; z9V;iWKD%D;8^@U2Ru1^o-yiR7MoYTo@9^35Sq?yeKv6mL`>YZISn)Y}h)J~W(C1|g z-$ec&(LBKtbqj6}a~$$0>VdM)Qk?%%veW16*}kq5-0J;6g|a>uP4fVPIiDx{9_|X} zoRO+L_YhTaP=do)BU#Yav+-P5eYx{RMDw~spYv$G&(yO43FT_9U z3h;oK46*1QerzNN>J6NnEDeL5lzAx}i4BkN;Rq^4>|uf$Mx>Y=1i$82D2XVEs6iOP zaZ*p@ugQ_IYasn@@FS5BQO#i8B+zqya1oTH!>#=qQOMVL0E=*oozkAIq=r8vD$0>6 zv{2QSAwye5egvfLkt#~JHMAqcPa3ak<4^aU<(m%f;hP$p z_Ve$&&F?+LpF7pfw>FTn>58ol=(a5liC`Sv2u5NG4}MZ1EM_PIMj}9|Vi8jo8yyUT zisY|}A@#^LUQ_`N1_3(gONn+mqPh+dj3vPf^*n)DU{WO|I2`6jL9T0irSlEd6AZZ;b&&$bQnEI5NtpDe1F1zk4e&XrQRMNX%#PDZM#yNp5Ln9j|5W3i=ElaSkG7KOq?wt|N2h;HX*%I?ODpZD zP2Puftr_^Zsb8s$SMIvpUiSq0yYke&Jx_c2UHyTl_U>tV+PjBdHb3p{jyv$M*4wY_ z#`jAjC<5*%c%raLK8RbdF>X(fUKNKM;;!_yNJ+2;WLgPo1a`cHt7GXaZ)pYs9y@#f z1ciVm1q_xEX0ip)Hi~J~3>VUf5Eu@o_OKj@MP))9f|gT9DM4n9fgbh)+-fETknT~E zX;#3t`A5KikP3?2j)(Fwl28Ch;iKV{W&)EYcaaM0g;Tv<-U7m)3nQQ%L*Z#B9^M{U1;K`o!J#e>k1k1V{)1;po)SMDwD9|IoqDcD`}; zV`tItJ*A5t{zDJ{M#1fppO(zo=a1d7r7AP9a=~*d?yBF?ueUSe_^9MD_rpJ;Oe5ullGwjEn;?FHi z(Vi86de8H%=-~JHg{0?1+uWgF4Za=tMIe=bc+$LNbxvEStTRO~8Wl}8PXvuNV{A0BRDinakKyUH}rCg5aOSsEw5f9Sx;{p<((9j92!e=`}7t|27} zW(9^|9()LuG(qVUpoc74*7ZL~mQ`wm`T=t2$toP7zBsltTzHbbHG}E39JQgir~uab0Z8PHDypS z!vHmbdMO84m$Slv@6acRr^L4%I4h_gRtjGBUsw>;k1(lRE> zVWMTJ&^?mzIq=j248y#o1qmO6%TFAHQ+Zds)O#5IOK1sHp~Qz8;fdhvj-URAEydoJIsj zNuX|66zYz1#P`a_3)2vlPLHln;;vhKd)htD-{Nn@9k*KQ>gsOcujN)+n;CA!js3Uc zHmbqF09}e3`9}FH8m8($M!FyfFK)WUUyd8E?4smPob{qT6uqUfl1jAmtyo)@!N=lie zn|?V;t~jD`;q6LC0uT~tOi8~8J2rxrVOcQ>X_eeN9 z{=`WTEZl%$xo}6Sux8Twg&6{S+3HH>J#o)^dAYDURk(A~`tc?{<*a|jI8?TenO1y2<; zCYSQmgGF+CW=eiAo^fy0rrh<5?gI<%1M}*g3n_Oyw0G^EIp14{etwA3IclOkG4$7` z5{DPuJMZj3F0`6s$}!_lR3}Pjjk9}a`~K$O?C!ZvFgxb?!DP{)-&$Kfw*fOev@;k_ z^K8jn(OYG2md!W+Er_o}N%qhm9#%5WV(2e2Q=g#WZsoU>(vjfSN{m5 zhRpwa-_E!ilq$^eU)VcanYTP$JJ{c_W~A@6I=U)N?_1nm<)-(`Ib5&VU&J7#s_wty zB9{^pvLp;qb5K^s9A#A{tnmzGg)3SrXQ;9=<|r!@#1`p;9)htHs?3u$qIKaFWkG+y zpCquWB5u`lh38xo2i0ZK+kqs2!dVrU8S!R>c(cTt8e;HBE9=~=Bz{CW3R7f&Ak!mz z8yjoX@u+kJzXt^CR>5kWV0xM3(2q~%uN6c$nF$UuAKHcJ zXA-=#-vr;dQS%V5)vy@Ha0Y0AO1B(ATgWliH);l1Arh`8u5ZmFOFs7vYz?(PbDw72}7CqG;daD2S#y=}WlKfe5sUZf9%ut(D z48eXoQssw1CCM;L?br4;zVqwPo}K^f*}vzlp2sO=!1&%ueorI-p zL^54#sz1cXPZQERBEy-_V?y~sm~TYhK(z0=-phx=eA6C&Ut=SG#+$D6S>iAGcpi<@ zfobiRBhW7K5}wlRei)bdGS}?(v=OUuG$zQ)QeP`EMMmYP@Dfy9dCgQ%X&kL1PvN;{ zPB)YG5Gs^&mlU$H(^4rdk({{`D!+u>U*bD_VNEsfiNq!07bySsF5M~XTD|M`9)lh;h8=8UI(8~ZnABx_b3@tD(1X{GX# zxbs%yMu4%7GRzp3jl4Z)TOme)0tRJqMg)q31X zD&USjikIkj4SGpR^blW9I{wYeYI*wgfY8-*{sP@L)D~a6W8fqUV0REdn8po;V)7sf zW{F1Nts;Z?%8qeob$B|GP9{41H~3Tj3lcceiDTx|(&{P4B)jY`PHw833(WbFCCw@K zexm66N(%9>dV zE_W4S1G$R-kI8Jaf6g(RYF3Ki;Vqf)v$^Q}AACV%jA%PaOxnB6*qOI(ntt=rd6OZdlwnWtrU6UM%>Dv4| z);76kku6263tOIIOKI`M_C?#ad$w(}Rm%Xpw&!g#)wAs%S!D?cD>8PP1)IL~3Ruvk!;`le>|j`w%jQPz93G z@TsO8Z^`P`qWLThui~26)s5DZ-Qs9Ca|q^DrbV#89JR*&5lZ{G9tgY#!Nw~1a+r|( zld+(rekE);9p4*}hGm3CD)3G4J42F)m}fvLF_iW=1Zey(-?)=PGLavYNra(29bP;d z0}oEz8e>ubQQ8EsWn#hW`HLjf7(vYl-wqI`#*Ls791J4HYP^LP^%!J(Iv^R^5*9zP zl!-kaM*OuHhR*~M;|j*2NFhWy5Vy#Qpn5Vk$hT)=$TK_wh<=QL2HNS)7bPYSA>0DB z)3F#>ygaZR4eTJ!8=y*pA`wmzN%IO(5pNO<+eJ7m$OuT-7f&{dlJ-GRa)PdTUcq}I zh#(GE5OD$QdjzhEaRo0f`GXkhxP4Iai!cJnTT&MSPk89Gg9C#=w5FmA#X>w8Q^q?I zBwHD=f+4sPHhmqZ5x3KZ5dP895+j@hPolR8bk0O?6PHN5pk@8zU7Ep5q6$ffem9~M z$5B=O4@khTt3c5*HY^ts2c?kwukbW&6=Wh0Su(})ukiRYk_GgKW&N-SYAxy9G21tL zVy&vps~cM2zDaZ9oi-o=SNkP8X009vNt`A7 z^mhmSo7LmcR+&*i&lBvzT@k5b}&r*WFbkFT3X^; ziC}8h1tfm)5u|R(Oix~i{4;c=nbSYMkjY1pJho?_C&79`~n;15>t1ZpFyiwtie# zJQMig!O6~LXZ}pz%!$OIWI^?Boi%XVlt>0xuJ*hahGzVWw$1l!n-hb%Ivw3?Sz$SD z>#`Mo8%o{nxM!{Y!pc}Zl&flEhnmTHMJq@46fvTW#<1j!rxGZf#K~ z$GyjKxPH&&=-g_1k9Tz0P4CqfcX6ioIS$vFbzp!@`M`jeg}>D;!cz?-7xdTpCHx>D zb`o9rBsqwm1qJwaJ(DMyc$N+~PkXjILSig{h{N`tQ+xRI4c`GHlJ7W+2j@C^>LK_o zN~l*xck#pRU`Ql!6ee#F&31h+kR(LMW0#8U(1g4%H5thD1w1ZAv#Hp&QW3guo>&gN{*SVCl#4yC;B% zh@ed48Z?eWx4{R^`p(r>FXwQ?|Ig(H{nFkvkzu~CQ{Dea$ zMu%36pD)$p*Km?l-$i~G7gl?5t#R~`=E!*_A0{nTl1i39P^LDpMI3}XAHVIu*Cu)y z!}&QWMNqW-Hc8$f$z_rpA;~$CbdcoVkc5OFrETp7az*BeQt#_U0onFxDL{5G>5X)c z-Uty$RdJf3vbK&!0x_@w@(0KunnL*!2`FBJ;d9nuU>|H|44zMzl1~`NCyb5$&;Nw6 zf5H@h%2Xwps^3{`&y77h_S`eiJ~Jb~Joe()%g?;{%&dHS?DetR&%FN3{LR$PM^mNk zDa)}5cFE?LE}bf!E}tr&2_(*>Y`Z4ROYm)9oVqyOKh>YuF>9GCPq_|E*nel)KEbY7 z890M|GY6B7@(C+YI1%Wg2=-|R3Py + Title + JUL 28, 2025 · 19 MIN + + + The previous version looked for a thumbnail `alt=` attribute and a + Title-Case date. The redesign has neither — no feature images on the index + and uppercase abbreviated months — so it matched nothing and silently + returned an empty list for weeks. + """ page = fetch(BLOG_URL) skip = ("/blog/tag/", "/blog/author/") - anchors = [ - m for m in re.finditer(r']+href="(/blog/[a-z0-9][a-z0-9-]*)"', page) - if not m.group(1).startswith(skip) - ] - seen, articles = set(), [] - for idx, m in enumerate(anchors): - slug = m.group(1) - if slug in seen: + articles, seen = [], set() + + row_re = re.compile( + r']+href="(/blog/[a-z0-9][a-z0-9-]*)"[^>]*>(.*?)' + r'.{0,400}?]*class="meta"[^>]*>(.*?)', + re.S, + ) + for m in row_re.finditer(page): + slug, raw_title, raw_meta = m.group(1), m.group(2), m.group(3) + if slug.startswith(skip) or slug in seen: continue - seen.add(slug) - stop = len(page) - for nxt in anchors[idx + 1:]: - if nxt.group(1) != slug: - stop = nxt.start() + + title = htmllib.unescape(re.sub(r"<[^>]+>", "", raw_title)).strip() + meta_text = htmllib.unescape(re.sub(r"<[^>]+>", " ", raw_meta)) + meta_text = re.sub(r"\s+", " ", meta_text).strip() + + # "JUL 28, 2025" — uppercase, abbreviated month. + date_m = re.search(r"([A-Za-z]{3,}) (\d{1,2}), (\d{4})", meta_text) + if not (title and date_m): + continue + month, day, year = date_m.groups() + date = None + for fmt in ("%b %d %Y", "%B %d %Y"): + try: + date = datetime.datetime.strptime(f"{month.title()} {day} {year}", fmt) break - window = page[m.start():stop] - title_m = re.search(r'alt="([^"]+)"', window) - stripped = re.sub(r"<[^>]+>", " ", window) - stripped = htmllib.unescape(re.sub(r"", "", stripped)) - date_m = re.search(r"([A-Z][a-z]+ \d{1,2}, \d{4})", stripped) - mins_m = re.search(r"(\d+)\s*min", stripped) - views_m = re.search(r"([\d,]+)\s*views", stripped) - if not (title_m and date_m): + except ValueError: + continue + if date is None: continue - date = datetime.datetime.strptime(date_m.group(1), "%B %d, %Y") - meta = f"{mins_m.group(1)} min read" if mins_m else ( - f"{views_m.group(1)} views" if views_m else "" - ) + + mins_m = re.search(r"(\d+)\s*MIN", meta_text, re.I) + seen.add(slug) articles.append({ - "title": htmllib.unescape(title_m.group(1)), + "title": title, "url": f"{SITE}{slug}", "date": date, - "meta": meta, + "meta": f"{mins_m.group(1)} min read" if mins_m else "", }) return articles @@ -124,8 +158,9 @@ def update_feed(text): try: articles = articles_from_rss() except Exception as exc: - print(f"feed: rss unusable ({exc}); falling back to html") + print(f"::warning title=nightly FEED rss::rss unusable ({exc}); falling back to html") if not articles: + print("::warning title=nightly FEED fallback::rss yielded nothing; scraping /blog") articles = articles_from_html() dated = [a for a in articles if a["date"]] dated.sort(key=lambda a: a["date"], reverse=True) @@ -188,22 +223,41 @@ def bump_entropy(): ) +def warn(label, exc): + """Fail-safe, but never quiet. + + Every section still degrades to 'leave it alone and exit 0' — a broken + fetch must not wedge the profile. What changed is visibility: a section + that stops updating now announces itself as a GitHub Actions warning + annotation, which surfaces in the run summary instead of scrolling past in + a log nobody opens. The FEED table sat frozen for 37 days precisely + because a silent skip and a successful run looked identical. + """ + print(f"::warning title=nightly {label} skipped::{label} left untouched — {exc}") + print(f"{label}: left untouched ({exc})", file=sys.stderr) + + def main(): with open(README) as fh: text = fh.read() + failures = 0 for label, fn in (("FEED", update_feed), ("STATS", update_stats), ("FEATURED", update_featured)): try: text = fn(text) print(f"{label}: updated") except Exception as exc: - print(f"{label}: left untouched ({exc})") + failures += 1 + warn(label, exc) with open(README, "w") as fh: fh.write(text) try: bump_entropy() print("ENTROPY: regenerated") except Exception as exc: - print(f"ENTROPY: left untouched ({exc})") + failures += 1 + warn("ENTROPY", exc) + if failures: + print(f"::warning title=nightly incomplete::{failures} of 4 sections did not update") if __name__ == "__main__":