(#d3sbkoq) i can see your twts here: https://watcher.sour.is/?uri=https://eapl.me/tw.txt

@david@collantes.us.. i see this one but it says its dead. https://watcher.sour.is/?uri=https://ferengi.one/twtxt.txt

(#d3sbkoq) it adds users by finding them in feeds mentioning or following. Your URL is already added.

Hmm i am not sure how you got the URL with users at the end..

Hi, So i made a little MVP registry crawler tool for twtxt. It now has a basic UI to play with. It has a somewhat full history back to about 2018-ish. Plus some interesting bits that were timestamped to earlier.

Find it here: https://watcher.sour.is

Code base is found here: https://git.sour.is/sour-is/xt

(#4bz3ama) probably this one https://yarn.girlonthemoon.xyz/user/kat/twtxt.txt

@kat@yarn.girlonthemoon.xyz

(#4bz3ama) yeah its from Yarn.. should be $VERSION@$COMMIT and its supposed to be replaced with the actual version and commit.

(#ywytxua) @lyse@lyse.isobeef.org this is 1990’s certified approved

(#pv75gxq) @bender@twtxt.net nope.. its causing OOM issues currently.

(#gyneajq) Registry format is its own thing. It takes the regular feed and appends nick \t uri \t to it. Its something that existed before yarn got big. There is still a bit of work but I will put together a ui for it to make it easier to view and navigate.

(#gyneajq) @eapl.me@eapl.me I am currently working on Implementing a registry that is also a crawler. It finds any feeds that are mentioned or in the follows header.

https://watcher.sour.is/api/plain/twt

https://watcher.sour.is/api/plain/users

I think @prologic@twtxt.net is also working on one.

(#zrsxk3a) hmm i need to start storing feed preambles so i can capture metadata like that

Interesting.. so running into an issue where queries only return a partal set of rows if i run in a docker image built from scratch. i have to add the debian root image for it to work. I wonder what file is missing that the root has?

(#kb25goa) Hmm I think I can come up with some kind of heuristic.. Maybe if the feed is requested and hasn’t updated in the last few mins it adds to the queue. So the next time it will be fresh.

(#okqf5bq)

(#okqf5bq) Just flip it left to right

(#qeqz7tq) @movq@www.uninformativ.de ✅

(#gkqvwra) Hmm so looking at the swagger of the registry spec client it seems to just take a “page”.. That seems worse than doing an offset. Lol.

https://github.com/DracoBlue/twtxt-registry/blob/master/src/swagger.json

(#srxmgka) An AI clock is correct twice a day.

I’m not much a fan of registry limit/offset paging. I think I prefer the cursor/count method. And starting at zero for first and max for latest.

(#4ro37ha) So smooth. love it!

(#rf27ppa) @doesnm@doesnm.p.psf.lt haha its not coming back. he talked of a stand alone thing like feeds. but not in yarnd

(#bqhddgq) hmm @prologic@twtxt.net how did replying to lyse double up here?

(#kb25goa) if it hasn’t updated in a while so i put the request rate to once a week it will take some time before i see an update if it happens today.

I need to figure out a way to back off requests to feeds that don’t update often.

(#zcabhya) I need to import my yarn cache. It’s sitting at about 1.5G in registry format. That should make things interesting…

(#zcabhya) neat! my watcher is currently sitting at about 75 MB following over 1500 feeds. only about 200 are currently somewhat active.

-rw-r--r--. 1 xuu  xuu   69M Mar 25 20:46 twt.db
-rw-r--r--. 1 xuu  xuu   32K Mar 25 21:34 twt.db-shm
-rw-r--r--. 1 xuu  xuu  5.6M Mar 25 21:34 twt.db-wal

sqlite> select state, count(*) n from feeds group by 1;
hot|7
warm|8
cold|183
frozen|743
permanantly-dead|857

(#bqhddgq) you just haven’t read this yet.