genaforvena/watching_u_watching

AI Summary: The task requires summarizing the provided project and issue descriptions, which detail a methodology for detecting systematic bias in institutions (using paired testing and data analysis) and a proposal for auditing Large Language Models (LLMs) to identify training data fingerprints using cryptohontology techniques. The summary should concisely explain the requirements of both the project and the issue, including the proposed audit categories and methods for the LLM audit.

AI Summary: The task involves replicating a study that extracts Personally Identifiable Information (PII) from the GPT-2 language model using novel text perturbation techniques. This requires developing a Python script to systematically perturb input prompts, feed them to the GPT-2 model, and analyze the output for the presence of specific, known PII strings from a previous study. The goal is to validate a new method for probing AI systems for vulnerabilities.

AI Summary: The task is to design and implement a Python script that conducts an automated bias audit of Google's Gemini Flash large language model (LLM). The audit will involve sending pairs of prompts to the LLM: one clean and one with randomly introduced typos. The script will then analyze the LLM's responses to these prompts, measuring differences in response length, sentiment, refusal rate, and latency to detect potential bias related to typographical errors. The audit must adhere to ethical and legal guidelines, using only synthetic data and respecting rate limits.

AI Summary: The task involves creating a policy brief document by copying existing content from a markdown file (`THE_MACHINERY_OF_ACCOUNTABILITY.md`), adding a short 10-line addendum detailing the implementation of a Wizard-of-Oz reverse study, and saving it to a specified location (`policy/human_attribution_policy_brief.md`). The document should adhere to specific formatting and include internal link checks.

AI Summary: The task is to expand an existing bias detection framework to include code generation and automated testing. This involves creating a new probe type for code generation, generating paired prompts (one with correct English, one with injected errors), collecting code responses from LLMs, implementing automated testing for the generated code, analyzing the results using objective metrics (test pass rate, functionality score), and integrating these findings into the existing bias analysis component. The goal is to quantitatively measure how linguistic variations in prompts affect the functionality of LLM-generated code.

AI Summary: The task is to analyze a project proposal and a related issue proposing a correspondence study to detect gender bias in literary agent queries. The analysis should provide a concise summary of the requirements for the correspondence study and rate the task's fun and complexity levels.

AI Summary: The task involves researching existing adversarial prompt techniques for Large Language Models (LLMs), adapting these techniques for integration into the 'watching_u_watching' framework for detecting AI biases beyond simple demographic factors, and documenting the proposed methodology with examples of adversarial prompts and response classification.

AI Summary: The task involves expanding the scope of the 'Watching U Watching' project to audit the bias detection tools themselves. This requires researching and selecting target systems (AI fairness tools), running dual evaluations (comparing the project's paired testing with standard fairness tool analysis), and publishing the results, including mismatches and explanations. The project also needs to address its own methodological limitations and biases, and develop a standardized comparison metric. Collaboration with researchers, legal experts, and ESG specialists is needed.

AI Summary: The task is to analyze the provided project description and issue details, which describe a system for detecting systematic discrimination using automated methods and propose integrating AI agents to enhance this system. The goal is to summarize the requirements for integrating AI agents into the bias detection system, focusing on ethical considerations and potential use cases, and then rate the task's fun and complexity.

AI Summary: The task involves replicating a study on ethnic discrimination in the German job market using an automated system. This requires identifying a suitable automated job application system, creating paired applications with German and Turkish-sounding names (without using LLMs), submitting them, and analyzing the response rates while rigorously redacting all PII from responses. The goal is to quantify any observed disparities, mirroring the original study's methodology.

AI Summary: The task is to design and execute an experiment using automated paired testing to detect and quantify potential systemic biases in a simulated Berlin housing market. The experiment will use a conversational AI model to simulate market responses to various user personas, with the goal of identifying biases related to perceived gender and ethnicity. The results will be analyzed using Fairlearn to measure disparities and determine statistical significance. The entire process must adhere to strict ethical guidelines, ensuring no harm to real individuals or organizations and avoiding the use of personally identifiable information.

AI Summary: The task is to assess the feasibility and potential of adapting the open-source project 'watching_u_watching' to comply with and enhance NYC Local Law 144, which mandates bias audits for AI in hiring. This involves evaluating how the project's methodology of automated paired testing can be tailored to generate probes specific to NYC employment contexts, report metrics required by the law, and ensure ethical and legal compliance. The goal is to propose concrete next steps for integrating 'watching_u_watching' into the LL144 compliance landscape.

AI Summary: The task is to analyze a project aiming to detect systematic discrimination using automated methods and then summarize the key challenges and proposed solutions outlined in a specific issue related to auditing email-based job applications under NYC Local Law 144. This involves understanding the project's methodology, the complexities of inferring AEDT usage from email responses, managing email accounts at scale, ethical considerations, and data analysis nuances.

AI Summary: The task is to analyze how the open-source project 'Watching U Watching,' designed to detect systematic bias in institutional processes, can be adapted to comply with and contribute to fulfilling the requirements of Brazil's AI Act (Bill No. 2338/2023), specifically focusing on bias mitigation, transparency, explainability, and Algorithmic Impact Assessments (AIAs) in the context of employment decisions. This involves discussing how the project's data collection and analysis can support AIAs, how to address the 'right to explanation,' navigating legal nuances of the Brazilian Act, developing compliance-focused features and reports, and fostering community engagement in Brazil.

AI Summary: The task is to analyze a project proposal for an automated bias detection system and an associated GitHub issue proposing the integration of African AI ethics and data diversity principles from the Masakhane initiative. The goal is to summarize the project's requirements and rate the task's fun and complexity.

AI Summary: The task is to analyze a project aiming to detect systematic discrimination using automated methods and an associated issue discussing the application of the project's methodology to the Nigerian context, considering data localization and cultural nuances. The goal is to summarize the project and issue, and rate the fun and complexity of analyzing them.

AI Summary: The task is to analyze a project aiming to detect systematic discrimination using automated methods and an associated issue proposing to leverage the project's findings to audit tech firms' ethical AI claims within ESG frameworks. This involves discussing how to map the project's bias detection metrics to ESG reporting standards, identify verifiable claims, develop suitable reporting formats, highlight gaps in current ESG frameworks, and prevent 'ethics-washing'.

AI Summary: The task involves designing and executing an experiment to detect bias in conversational AI using automated paired testing. This includes creating diverse prompts with varied user personas (based on gender and ethnicity), sending these prompts to a conversational AI model, collecting and quantifying the responses, and analyzing the results using Fairlearn to identify statistically significant biases. The project leverages existing social science methodologies and aims for a scalable, efficient, and transparent approach.

AI Summary: The task involves executing a pilot audit to detect bias in NYC-based job postings using an automated email system. This includes identifying target job postings, generating synthetic applications, submitting them, monitoring responses, analyzing the data for bias according to NYC Local Law 144 guidelines, and generating a transparent report of the findings. The report will be publicized to raise awareness and solicit feedback.