kyle/sift
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

p1: /v2/scrape, /v1/fetch, auth, CLI, contract tests

Browse Source 
Single Fetcher.Fetch on the Guard that holds DenyPrefixes. skipTlsVerification
is ignored. No 502. keys add writes the secret only to --out. Release binary
has no -test-allow-prefix.
This commit is contained in:
kyle
2026-09-22 21:54:01 -07:00
parent 6cb206fb27
commit 9444a35322
311 changed files with 238557 additions and 92 deletions
Download Patch File Download Diff File Expand all files Collapse all files
vendor/github.com/JohannesKaufmann/dom/.gitignore
Generated
+22
View File
@@ -0,0 +1,22 @@
# If you prefer the allow list template instead of the deny list, see community template:
# https://github.com/github/gitignore/blob/main/community/Golang/Go.AllowList.gitignore
#
# Binaries for programs and plugins
*.exe
*.exe~
*.dll
*.so
*.dylib
# Test binary, built with `go test -c`
*.test
# Output of the go coverage tool, specifically when used with LiteIDE
*.out
# Dependency directories (remove the comment below to include it)
# vendor/
# Go workspace file
go.work
go.work.sum
vendor/github.com/JohannesKaufmann/dom/LICENSE
Generated Vendored
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2024 Johannes Kaufmann
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
vendor/github.com/JohannesKaufmann/dom/README.md
Generated Vendored
+277
View File
@@ -0,0 +1,277 @@
# dom
[![Go Reference](https://pkg.go.dev/badge/github.com/JohannesKaufmann/dom.svg)](https://pkg.go.dev/github.com/JohannesKaufmann/dom)
Helper functions for "net/html" that make it easier to interact with `*html.Node`.
🚀 [Getting Started](#getting-started) - 📚 [Documentation](#documentation) - 🧑‍💻 [Examples](/examples/)
## Installation
```bash
go get -u github.com/JohannesKaufmann/dom
```
> [!NOTE]
> This "dom" libary was developed for the needs of the [html-to-markdown](https://github.com/JohannesKaufmann/html-to-markdown) library.
> That beeing said, please submit any functions that you need.
## Getting Started
```go
package main
import (
"fmt"
"log"
"strings"
"github.com/JohannesKaufmann/dom"
"golang.org/x/net/html"
)
func main() {
input := `
<ul>
<li><a href="github.com/JohannesKaufmann/dom">dom</a></li>
<li><a href="github.com/JohannesKaufmann/html-to-markdown">html-to-markdown</a></li>
</ul>
`
doc, err := html.Parse(strings.NewReader(input))
if err != nil {
log.Fatal(err)
}
// - - - //
firstLink := dom.FindFirstNode(doc, func(node *html.Node) bool {
return dom.NodeName(node) == "a"
})
fmt.Println("href:", dom.GetAttributeOr(firstLink, "href", ""))
}
```
## Node vs Element
The naming scheme in this library is:
- "Node" means `*html.Node{}`
- This means _any_ node in the tree of nodes.
- "Element" means `*html.Node{Type: html.ElementNode}`
- This means _only_ nodes with the type of `ElementNode`. For example `<p>`, `<span>`, `<a>`, ... but not `#text`, `<!--comment-->`, ...
For most functions, there are two versions. For example:
- `FirstChildNode()` and `FirstChildElement()`
- `AllChildNodes()` and `AllChildElements()`
- ...
## Documentation
[![Go Reference](https://pkg.go.dev/badge/github.com/JohannesKaufmann/dom.svg)](https://pkg.go.dev/github.com/JohannesKaufmann/dom)
### Attributes & Content
You can get the attributes of a node using `GetAttribute`, `GetAttributeOr` or the more specialized `GetClasses` that returns a slice of strings.
For matching nodes, `HasID` and `HasClass` can be used.
If you want to collect the #text of all the child nodes, you can call `CollectText`.
```go
name := dom.NodeName(node)
// "h2"
href := dom.GetAttributeOr(node, "href", "")
// "github.com"
isHeading := dom.HasClass(node, "repo__name")
// `true`
content := dom.CollectText(node)
// "Lorem ipsum"
```
---
### Children & Siblings
You can already use `node.FirstChild` to get the first child _node_. For the convenience we added `FirstChildNode()` and `FirstChildElement()` which returns `*html.Node`.
To get all direct children, use `AllChildNodes` and `AllChildElements` which returns `[]*html.Node`.
- `PrevSiblingNode` and `PrevSiblingElement`
- `NextSiblingNode` and `NextSiblingElement`
### Find Nodes
Searching for nodes deep in the tree is made easier with:
```go
firstParagraph := dom.FindFirstNode(doc, func(node *html.Node) bool {
return dom.NodeName(node) == "p"
})
// *html.Node
allParagraphs := dom.FindAllNodes(doc, func(node *html.Node) bool {
return dom.NodeName(node) == "p"
})
// []*html.Node
```
- 🧑‍💻 [Example code, find](/examples/find/main.go)
- 🧑‍💻 [Example code, selectors](/examples/selectors/main.go)
---
### Get next/previous neighbors
What is special about this? The order!
If you are somewhere in the DOM, you can call `GetNextNeighborNode` to get the next node, even if it is _further up_ the tree. The order is the same as you would see the elements in the DOM.
```go
node := startNode
for node != nil {
fmt.Println(dom.NodeName(node))
node = dom.GetNextNeighborNode(node)
}
```
If we start the `for` loop at the `<button>` and repeatedly call `GetNextNeighborNode` this would be the _order_ that the nodes are _visited_.
```text
#document
├─html
│ ├─head
│ ├─body
│ │ ├─nav
│ │ │ ├─p
│ │ │ │ ├─#text "up"
│ │ ├─main
│ │ │ ├─button *️⃣
│ │ │ │ ├─span 0️⃣
│ │ │ │ │ ├─#text "start" 1️⃣
│ │ │ ├─div 2️⃣
│ │ │ │ ├─h3 3️⃣
│ │ │ │ │ ├─#text "heading" 4️⃣
│ │ │ │ ├─p 5️⃣
│ │ │ │ │ ├─#text "description" 6️⃣
│ │ ├─footer 7️⃣
│ │ │ ├─p 8️⃣
│ │ │ │ ├─#text "down" 9️⃣
```
If you only want to visit the ElementNode's (and skip the `#text` Nodes) you can use `GetNextNeighborElement` instead.
If you want to skip the children you can use `GetNextNeighborNodeExcludingOwnChild`. In the example above, when starting at the `<button>` the next node would be the `<div>`.
The same functions also exist for the previous nodes, e.g. `GetPrevNeighborNode`.
- 🧑‍💻 [Example code, next basics](/examples/next_basics/main.go)
- 🧑‍💻 [Example code, next inside a loop](/examples/next_loop/main.go)
---
### Remove & Replace Node
```go
if dom.HasClass(node, "lang__old") {
newNode := &html.Node{
Type: html.TextNode,
Data: "🪦",
}
dom.ReplaceNode(node, newNode)
}
for _, node := range emptyTextNodes {
dom.RemoveNode(node)
}
```
- 🧑‍💻 [Example code, remove and replace](/examples/remove_replace/main.go)
### Unwrap Node
```text
#document
├─html
│ ├─head
│ ├─body
│ │ ├─article *️⃣
│ │ │ ├─h3
│ │ │ │ ├─#text "Heading"
│ │ │ ├─p
│ │ │ │ ├─#text "short description"
```
If we take the input above and run `UnwrapNode(articleNode)` we can "unwrap" the `<article>`. That means removing the `<article>` while _keeping_ the children (`<h3>` and `<p>`).
```text
#document
├─html
│ ├─head
│ ├─body
│ │ ├─h3
│ │ │ ├─#text "Heading"
│ │ ├─p
│ │ │ ├─#text "short description"
```
For the reverse you can use `WrapNode(existingNode, newNode)`.
---
### RenderRepresentation
```go
import (
"fmt"
"log"
"strings"
"github.com/JohannesKaufmann/dom"
"golang.org/x/net/html"
)
func main() {
input := `<a href="/about">Read More</a>`
doc, err := html.Parse(strings.NewReader(input))
if err != nil {
log.Fatal(err)
}
fmt.Println(dom.RenderRepresentation(doc))
}
```
The tree representation helps to visualize the tree-structure of the DOM.
And the `#text` nodes stand out.
> [!TIP]
> This function could be useful for debugging & testcases.
> For example in [neighbors_test.go](/neighbors_test.go)
```text
#document
├─html
│ ├─head
│ ├─body
│ │ ├─a (href=/about)
│ │ │ ├─#text "Read More"
```
While the normal "net/html" [`Render()`](https://pkg.go.dev/golang.org/x/net/html#Render) function would have produced this:
```
<html><head></head><body><a href="/about">Read More</a></body></html>
```
- 🧑‍💻 [Example code, dom representation](/examples/dom_representation/main.go)
vendor/github.com/JohannesKaufmann/dom/attr.go
Generated Vendored
+68
View File
@@ -0,0 +1,68 @@
package dom
import (
"bytes"
"strings"
"golang.org/x/net/html"
)
func GetAttribute(node *html.Node, key string) (string, bool) {
for _, attr := range node.Attr {
if attr.Key == key {
return attr.Val, true
}
}
return "", false
}
func GetAttributeOr(node *html.Node, key string, fallback string) string {
for _, attr := range node.Attr {
if attr.Key == key {
return attr.Val
}
}
return fallback
}
func GetClasses(node *html.Node) []string {
val, found := GetAttribute(node, "class")
if !found {
return nil
}
return strings.Fields(val)
}
func HasID(node *html.Node, expectedID string) bool {
val, found := GetAttribute(node, "id")
if !found {
return false
}
return strings.TrimSpace(val) == expectedID
}
func HasClass(node *html.Node, expectedClass string) bool {
classes := GetClasses(node)
for _, class := range classes {
if class == expectedClass {
return true
}
}
return false
}
// - - - - //
func collectText(n *html.Node, buf *bytes.Buffer) {
if n.Type == html.TextNode {
buf.WriteString(n.Data)
}
for c := n.FirstChild; c != nil; c = c.NextSibling {
collectText(c, buf)
}
}
func CollectText(node *html.Node) string {
var buf bytes.Buffer
collectText(node, &buf)
return buf.String()
}
vendor/github.com/JohannesKaufmann/dom/change.go
Generated Vendored
+49
View File
@@ -0,0 +1,49 @@
package dom
import "golang.org/x/net/html"
func RemoveNode(node *html.Node) {
if node == nil || node.Parent == nil {
return
}
node.Parent.RemoveChild(node)
}
func ReplaceNode(node, newNode *html.Node) {
if node.Parent == nil || node == newNode {
return
}
node.Parent.InsertBefore(newNode, node)
node.Parent.RemoveChild(node)
}
func UnwrapNode(node *html.Node) {
if node == nil || node.Parent == nil {
return
}
// In each iteration, we once again grab the first child, since
// the previous first child was just removed.
for child := node.FirstChild; child != nil; child = node.FirstChild {
node.RemoveChild(child)
node.Parent.InsertBefore(child, node)
}
node.Parent.RemoveChild(node)
}
// WrapNode wraps the newNode around the existingNode.
func WrapNode(existingNode, newNode *html.Node) *html.Node {
if existingNode == nil || existingNode.Parent == nil {
return existingNode
}
existingNode.Parent.InsertBefore(newNode, existingNode)
existingNode.Parent.RemoveChild(existingNode)
newNode.AppendChild(existingNode)
return newNode
}
vendor/github.com/JohannesKaufmann/dom/dom.go
Generated Vendored
+88
View File
@@ -0,0 +1,88 @@
// dom makes it easier to interact with the html document.
//
// Node = return all the nodes
// Element = return all the nodes that are of type Element. This e.g. excludes #text nodes.
package dom
import "golang.org/x/net/html"
// AllNodes recursively gets all the nodes in the tree.
func AllNodes(startNode *html.Node) (allNodes []*html.Node) {
var finder func(*html.Node)
finder = func(node *html.Node) {
allNodes = append(allNodes, node)
for child := node.FirstChild; child != nil; child = child.NextSibling {
finder(child)
}
}
finder(startNode)
return allNodes
}
// - - - - - - - - - - - - - - - //
func AllChildNodes(node *html.Node) (children []*html.Node) {
for child := node.FirstChild; child != nil; child = child.NextSibling {
children = append(children, child)
}
return children
}
// AllChildElements is similar to AllChildNodes but only returns
// nodes of type `ElementNode`.
func AllChildElements(node *html.Node) (children []*html.Node) {
for child := node.FirstChild; child != nil; child = child.NextSibling {
if child.Type == html.ElementNode {
children = append(children, child)
}
}
return children
}
// - - - - - - - - - - - - - - - //
func FirstChildNode(node *html.Node) *html.Node {
return node.FirstChild
}
func FirstChildElement(node *html.Node) *html.Node {
for child := node.FirstChild; child != nil; child = child.NextSibling {
if child.Type == html.ElementNode {
return child
}
}
return nil
}
// - - - - - - - - - - - - - - - //
func PrevSiblingNode(node *html.Node) *html.Node {
return node.PrevSibling
}
func PrevSiblingElement(node *html.Node) *html.Node {
for sibling := node.PrevSibling; sibling != nil; sibling = sibling.PrevSibling {
if sibling.Type == html.ElementNode {
return sibling
}
}
return nil
}
// - - - - - - - - - - - - - - - //
func NextSiblingNode(node *html.Node) *html.Node {
return node.NextSibling
}
// NextSiblingElement returns the element immediately following the passed-in node or nil.
// In contrast to `node.NextSibling` this only returns the next `ElementNode`.
func NextSiblingElement(node *html.Node) *html.Node {
for sibling := node.NextSibling; sibling != nil; sibling = sibling.NextSibling {
if sibling.Type == html.ElementNode {
return sibling
}
}
return nil
}
vendor/github.com/JohannesKaufmann/dom/dom_representation.go
Generated Vendored
+64
View File
@@ -0,0 +1,64 @@
package dom
import (
"bytes"
"fmt"
"strings"
"golang.org/x/net/html"
)
func writePipeChar(buf *bytes.Buffer, index int) {
if index == 0 {
return
}
buf.WriteString(strings.Repeat("│ ", index-1))
buf.WriteString("├─")
}
func writeNode(buf *bytes.Buffer, node *html.Node) {
name := NodeName(node)
buf.WriteString(name)
if len(node.Attr) != 0 {
buf.WriteString(" (")
}
for i, attr := range node.Attr {
buf.WriteString(fmt.Sprintf("%s=%q", attr.Key, attr.Val))
if i < len(node.Attr)-1 {
buf.WriteString(" ")
}
}
if len(node.Attr) != 0 {
buf.WriteString(")")
}
if name == "#text" {
buf.WriteString(fmt.Sprintf(" %q", node.Data))
}
}
// RenderRepresentation is useful for debugging.
// It renders out the *structure* of the dom.
func RenderRepresentation(startNode *html.Node) string {
var buf bytes.Buffer
var finder func(*html.Node, int)
finder = func(node *html.Node, index int) {
writePipeChar(&buf, index)
writeNode(&buf, node)
buf.WriteRune('\n')
for child := node.FirstChild; child != nil; child = child.NextSibling {
finder(child, index+1)
}
}
if startNode.Parent == nil {
finder(startNode, 0)
} else {
finder(startNode, 1)
}
return strings.TrimSpace(buf.String())
}
vendor/github.com/JohannesKaufmann/dom/find.go
Generated Vendored
+49
View File
@@ -0,0 +1,49 @@
package dom
import "golang.org/x/net/html"
func ContainsNode(startNode *html.Node, matchFn func(node *html.Node) bool) bool {
return FindFirstNode(startNode, matchFn) != nil
}
func FindFirstNode(startNode *html.Node, matchFn func(node *html.Node) bool) *html.Node {
nextFunc := UNSTABLE_initGetNeighbor(
FirstChildNode,
NextSiblingNode,
func(node *html.Node) bool {
// We should not get higher up than the startNode...
return node == startNode
},
)
child := startNode.FirstChild
for child != nil {
if matchFn(child) {
return child
}
child = nextFunc(child)
}
return nil
}
func FindAllNodes(startNode *html.Node, matchFn func(node *html.Node) bool) (foundNodes []*html.Node) {
nextFunc := UNSTABLE_initGetNeighbor(
FirstChildNode,
NextSiblingNode,
func(node *html.Node) bool {
// We should not get higher up than the startNode...
return node == startNode
},
)
child := startNode.FirstChild
for child != nil {
if matchFn(child) {
foundNodes = append(foundNodes, child)
}
child = nextFunc(child)
}
return foundNodes
}
vendor/github.com/JohannesKaufmann/dom/neighbors.go
Generated Vendored
+108
View File
@@ -0,0 +1,108 @@
package dom
import "golang.org/x/net/html"
// Warning: It is not meant to be called directly and may change signature from release to release!
func UNSTABLE_initGetNeighbor(
firstChildFunc func(node *html.Node) *html.Node,
prevNextFunc func(node *html.Node) *html.Node,
goUpUntilFunc func(node *html.Node) bool,
) func(*html.Node) *html.Node {
return func(node *html.Node) *html.Node {
// First look at the children
if child := firstChildFunc(node); child != nil {
return child
}
// Otherwise my prev/next sibling
if sibling := prevNextFunc(node); sibling != nil {
return sibling
}
for {
// Finally, continously go upwards until we find an element with a sibling
node = node.Parent
if node == nil {
// We reached the top
return nil
}
if goUpUntilFunc(node) {
// Don't go too far up...
return nil
}
sibling := prevNextFunc(node)
if sibling != nil {
return sibling
}
}
}
}
// - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - //
// - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - //
var goUpForever = func(node *html.Node) bool { return false }
var skipFirstChild = func(node *html.Node) *html.Node { return nil }
func GetPrevNeighborNode(node *html.Node) *html.Node {
return UNSTABLE_initGetNeighbor(
FirstChildNode,
PrevSiblingNode,
goUpForever,
)(node)
}
func GetPrevNeighborElement(node *html.Node) *html.Node {
return UNSTABLE_initGetNeighbor(
FirstChildElement,
PrevSiblingElement,
goUpForever,
)(node)
}
func GetPrevNeighborNodeExcludingOwnChild(node *html.Node) *html.Node {
return UNSTABLE_initGetNeighbor(
skipFirstChild,
PrevSiblingNode,
goUpForever,
)(node)
}
func GetPrevNeighborElementExcludingOwnChild(node *html.Node) *html.Node {
return UNSTABLE_initGetNeighbor(
skipFirstChild,
PrevSiblingElement,
goUpForever,
)(node)
}
// - - - - - - - - //
func GetNextNeighborNode(node *html.Node) *html.Node {
return UNSTABLE_initGetNeighbor(
FirstChildNode,
NextSiblingNode,
goUpForever,
)(node)
}
func GetNextNeighborElement(node *html.Node) *html.Node {
return UNSTABLE_initGetNeighbor(
FirstChildElement,
NextSiblingElement,
goUpForever,
)(node)
}
func GetNextNeighborNodeExcludingOwnChild(node *html.Node) *html.Node {
return UNSTABLE_initGetNeighbor(
skipFirstChild,
NextSiblingNode,
goUpForever,
)(node)
}
func GetNextNeighborElementExcludingOwnChild(node *html.Node) *html.Node {
return UNSTABLE_initGetNeighbor(
skipFirstChild,
NextSiblingElement,
goUpForever,
)(node)
}
vendor/github.com/JohannesKaufmann/dom/tags.go
Generated Vendored
+147
View File
@@ -0,0 +1,147 @@
package dom
import "golang.org/x/net/html"
// In order to stay consistent with v1 of the library, this follows
// the naming scheme of goquery.
// E.g. "#text", "div", ...
func NodeName(node *html.Node) string {
if node == nil {
return ""
}
switch node.Type {
case html.ErrorNode:
return "#error"
case html.TextNode:
return "#text"
case html.DocumentNode:
return "#document"
case html.CommentNode:
return "#comment"
case html.DoctypeNode:
// E.g. for `<!DOCTYPE html>` it would be "html"
return node.Data
case html.ElementNode:
// E.g. "div" or "p"
return node.Data
}
return ""
}
func NameIsInlineNode(name string) bool {
switch name {
case
"#text",
"a",
"abbr",
"acronym",
"audio",
"b",
"bdi",
"bdo",
"big",
"br",
"button",
"canvas",
"cite",
"code",
"data",
"datalist",
"del",
"dfn",
"em",
"embed",
"i",
"iframe",
"img",
"input",
"ins",
"kbd",
"label",
"map",
"mark",
"meter",
"noscript",
"object",
"output",
"picture",
"progress",
"q",
"ruby",
"s",
"samp",
"script",
"select",
"slot",
"small",
"span",
"strong",
"sub",
"sup",
"svg",
"template",
"textarea",
"time",
"u",
"tt",
"var",
"video",
"wbr":
return true
default:
return false
}
}
func NameIsBlockNode(name string) bool {
switch name {
case
"address",
"article",
"aside",
"blockquote",
"details",
"dialog",
"dd",
"div",
"dl",
"dt",
"fieldset",
"figcaption",
"figure",
"footer",
"form",
"h1", "h2", "h3", "h4", "h5", "h6",
"header",
"hgroup",
"hr",
"li",
"main",
"nav",
"ol",
"p",
"pre",
"section",
"table",
"ul":
return true
default:
return false
}
}
func NameIsHeading(name string) bool {
switch name {
case
"h1", "h2", "h3", "h4", "h5", "h6":
return true
default:
return false
}
}