W1CDN/ham-radio-licenses
1
0 Fork 0
Code Issues 133 Pull Requests Packages Projects Releases Wiki Activity

AE7Q new license counts #114

New Issue
Open
opened 2024-02-28 14:16:46 -06:00 by matt · 8 comments
matt commented 2024-02-28 14:16:46 -06:00
Copy link
Copy source

To track new licenses over time, can we scrape the table under "Retrieving license grants issued on"? Note the date in the URL for easy downloading.

https://www.ae7q.com/query/list/ProcessDate.php?DATE=2024-2-02

This goes back to one of the W5YI Report comments that the only way to really measure growth is to measure the number of new hams.

To track new licenses over time, can we scrape the table under "Retrieving license grants issued on"? Note the date in the URL for easy downloading. https://www.ae7q.com/query/list/ProcessDate.php?DATE=2024-2-02 This goes back to one of the W5YI Report comments that the only way to really measure growth is to measure the number of new hams.
W1CDN commented 2024-04-21 21:23:54 -05:00
Owner
Copy link
Copy source

Looks like new license info goes back to 1994-05-24, but renewals and callsign changes aren't being tracked: https://www.ae7q.com/query/list/ProcessDate.php?DATE=1994-05-24

Looks like renewals started being tracked on 1999-08-28: https://www.ae7q.com/query/list/ProcessDate.php?DATE=1999-08-28

Sometime in 2001, vanity callsigns added to that table.

Table structure seems consistent over time.

Looks like new license info goes back to 1994-05-24, but renewals and callsign changes aren't being tracked: https://www.ae7q.com/query/list/ProcessDate.php?DATE=1994-05-24 Looks like renewals started being tracked on 1999-08-28: https://www.ae7q.com/query/list/ProcessDate.php?DATE=1999-08-28 Sometime in 2001, vanity callsigns added to that table. Table structure seems consistent over time.
W1CDN added the data-new label 2024-09-16 11:44:22 -05:00
W1CDN commented 2024-09-21 13:07:08 -05:00
Owner
Copy link
Copy source

I summarized down to here so far. How should I structure the final CSV file?

Shared on fedi: https://mastodon.radio/@W1CDN/113176909232406655

I summarized down to here so far. How should I structure the final CSV file? Shared on fedi: https://mastodon.radio/@W1CDN/113176909232406655 ![](/attachments/e8f9c8aa-1d81-4623-bb94-fbdd3c177ead)
image.png
186 KiB
W1CDN commented 2024-09-21 13:14:45 -05:00
Owner
Copy link
Copy source

"License Status" is current status of that license as of query date, so if you query back a ways, you will see lots of expired licenses, which isn't indicative of what the status was on that date.

Looks like SYSGRT was introduced on https://www.ae7q.com/query/list/ProcessDate.php?DATE=2004-11-29.

"License Status" is *current* status of that license as of query date, so if you query back a ways, you will see lots of expired licenses, which isn't indicative of what the status was on that date. Looks like SYSGRT was introduced on https://www.ae7q.com/query/list/ProcessDate.php?DATE=2004-11-29.
W1CDN commented 2024-09-21 19:19:30 -05:00
Owner
Copy link
Copy source

Basics are here, data is in since 2024-08-01.
Need to add plot(s) to readme when I have more data.
81e5fbef6c

Basics are here, data is in since 2024-08-01. Need to add plot(s) to readme when I have more data. https://amiok.net/gitea/W1CDN/ham-radio-licenses/commit/81e5fbef6c12ba59337f063b36d534e6b44aa69c
W1CDN commented 2024-09-21 22:39:53 -05:00
Owner
Copy link
Copy source

Added one plot to readme, will download historical data over time, new data scraped daily as part of cron job.

Added one plot to readme, will download historical data over time, new data scraped daily as part of cron job.
W1CDN commented 2024-11-28 10:23:43 -06:00
Owner
Copy link
Copy source

This script wasn't working, but I think is fixed in 1b58d335f8. It needs to query the previous date, not the current date.

This script wasn't working, but I think is fixed in 1b58d335f8. It needs to query the previous date, not the current date.
W1CDN commented 2025-02-24 21:11:35 -06:00
Owner
Copy link
Copy source

The scraper isn't working perfectly, need to debug.

The scraper isn't working perfectly, need to debug.
W1CDN commented 2026-08-02 18:27:20 -05:00
Owner
Copy link
Copy source

Scraper is broken. There is a CDN in front now.

Attaching package: ‘dplyr’

The following objects are masked from ‘package:stats’:

    filter, lag

The following objects are masked from ‘package:base’:

    intersect, setdiff, setequal, union

Checking ARRL...Checking HamCall...Checking AE7Q...Error in open.connection(x, "rb") : cannot open the connection
Calls: read_html -> read_html.default
Execution halted
Scraper is broken. There is a CDN in front now. ``` Attaching package: ‘dplyr’ The following objects are masked from ‘package:stats’: filter, lag The following objects are masked from ‘package:base’: intersect, setdiff, setequal, union Checking ARRL...Checking HamCall...Checking AE7Q...Error in open.connection(x, "rb") : cannot open the connection Calls: read_html -> read_html.default Execution halted ```
Sign in to join this conversation.
No Branch/Tag Specified
Labels
Clear labels
bug
Something is not working
 data-fix
data-new
duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 wontfix
This won't be fixed
No labels data-new
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
W1CDN (Matt )
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: W1CDN/ham-radio-licenses#114