Evalgate good first issues (TypeScript)

New scorer refusal: refusal detector for safety evals about 2 months ago

help wanted good first issue

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

New scorer tool-call: tool-call format for agentic evals about 2 months ago

help wanted good first issue

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

New scorer word-count: output length bounds about 2 months ago

help wanted good first issue

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

New scorer json-nonempty: fail schema-valid-but-empty outputs about 2 months ago

help wanted good first issue

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

String-form tags silently dropped, --tags skips the case with no warning about 2 months ago

help wanted good first issue

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

llm-judge passes on unclamped scores, a 0-100 reply always passes about 2 months ago

help wanted good first issue

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

help wanted good first issue

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

Rubric criterion with invalid pattern regex throws and kills the whole run about 2 months ago

help wanted good first issue

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

compareRuns accepts negative/NaN tolerance about 2 months ago

help wanted good first issue

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

Scorer weight: Infinity passes validation and poisons aggregation about 2 months ago

help wanted good first issue

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

contains scorer with empty-string needle always passes about 2 months ago

help wanted good first issue

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

Suite validation accepts empty prompts and bogus message roles about 2 months ago

help wanted good first issue

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

Invalid numeric CLI flags silently fall back instead of erroring about 2 months ago

help wanted good first issue

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

temperature and maxTokens are plumbed through the provider but cannot be set by a suite 2 months ago

enhancement good first issue claimed

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

An invalid pattern in a JSON schema throws and kills the whole run 2 months ago

bug good first issue claimed

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

exact-match always collapses internal whitespace, and trim: false does not turn it off 3 months ago

good first issue

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

Docs: --concurrency, --junit and --degrade are missing from evalgate --help 3 months ago

documentation good first issue

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

evalgate list hardcodes the provider names instead of asking the registry 3 months ago

enhancement good first issue claimed

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

Cost is one hardcoded rate per provider, applied to every model and both token directions 3 months ago

bug enhancement help wanted

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

Provider calls have no timeout and no retry, so a 429 fails the build 3 months ago

bug enhancement help wanted

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

One provider or scorer error discards the entire run 3 months ago

bug enhancement help wanted

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

Deleting a failing case makes the gate green: removed cases are never regressions 3 months ago

bug enhancement help wanted

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

Bug: a run with zero cases reports passed and exits 0, so the gate silently disables itself 3 months ago

bug help wanted

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

--tags and --concurrency are ignored by baseline and compare 3 months ago

bug help wanted

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

AgentPostmortem/Evalgate

Open Issues Need Help