codex/prs/bolinfest/PR-2651.md

# PR #2651: [apply-patch] Fix lark grammar

- URL: https://github.com/openai/codex/pull/2651
- Author: dylan-hurd-oai
- Created: 2025-08-25 00:39:19 UTC
- Updated: 2025-09-02 02:20:23 UTC
- Changes: +22/-21, Files changed: 2, Commits: 2

## Description

## Summary
Fixes an issue with the lark grammar definition for the apply_patch freeform tool. This does NOT change the defaults, merely patches the root cause of the issue we were seeing with empty lines, and an issue with config flowing through correctly.

Specifically, the following requires that a line is non-empty:
```
add_line: "+" /(.+)/ LF -> line
```
but many changes _should_ involve creating/updating empty lines. The new definition is:
```
add_line: "+" /(.*)/ LF -> line
```

## Testing
- [x] Tested locally, reproduced the issue without the update and confirmed that the model will produce empty lines wiht the new lark grammar

## Full Diff

```diff
diff --git a/codex-rs/core/src/tool_apply_patch.lark b/codex-rs/core/src/tool_apply_patch.lark
new file mode 100644
index 0000000000..5aa41b0af7
--- /dev/null
+++ b/codex-rs/core/src/tool_apply_patch.lark
@@ -0,0 +1,19 @@
+start: begin_patch hunk+ end_patch
+begin_patch: "*** Begin Patch" LF
+end_patch: "*** End Patch" LF?
+
+hunk: add_hunk | delete_hunk | update_hunk
+add_hunk: "*** Add File: " filename LF add_line+
+delete_hunk: "*** Delete File: " filename LF
+update_hunk: "*** Update File: " filename LF change_move? change?
+
+filename: /(.+)/
+add_line: "+" /(.*)/ LF -> line
+
+change_move: "*** Move to: " filename LF
+change: (change_context | change_line)+ eof_line?
+change_context: ("@@" | "@@ " /(.+)/) LF
+change_line: ("+" | "-" | " ") /(.*)/ LF
+eof_line: "*** End of File" LF
+
+%import common.LF
diff --git a/codex-rs/core/src/tool_apply_patch.rs b/codex-rs/core/src/tool_apply_patch.rs
index 3868cf0e70..d4039fd192 100644
--- a/codex-rs/core/src/tool_apply_patch.rs
+++ b/codex-rs/core/src/tool_apply_patch.rs
@@ -8,6 +8,8 @@ use crate::openai_tools::JsonSchema;
 use crate::openai_tools::OpenAiTool;
 use crate::openai_tools::ResponsesApiTool;
 
+const APPLY_PATCH_LARK_GRAMMAR: &str = include_str!("tool_apply_patch.lark");
+
 #[derive(Serialize, Deserialize)]
 pub(crate) struct ApplyPatchToolArgs {
     pub(crate) input: String,
@@ -29,27 +31,7 @@ pub(crate) fn create_apply_patch_freeform_tool() -> OpenAiTool {
         format: FreeformToolFormat {
             r#type: "grammar".to_string(),
             syntax: "lark".to_string(),
-            definition: r#"start: begin_patch hunk+ end_patch
-begin_patch: "*** Begin Patch" LF
-end_patch: "*** End Patch" LF?
-
-hunk: add_hunk | delete_hunk | update_hunk
-add_hunk: "*** Add File: " filename LF add_line+
-delete_hunk: "*** Delete File: " filename LF
-update_hunk: "*** Update File: " filename LF change_move? change?
-
-filename: /(.+)/
-add_line: "+" /(.+)/ LF -> line
-
-change_move: "*** Move to: " filename LF
-change: (change_context | change_line)+ eof_line?
-change_context: ("@@" | "@@ " /(.+)/) LF
-change_line: ("+" | "-" | " ") /(.+)/ LF
-eof_line: "*** End of File" LF
-
-%import common.LF
-"#
-            .to_string(),
+            definition: APPLY_PATCH_LARK_GRAMMAR.to_string(),
         },
     })
 }
```

## Review Comments

### codex-rs/core/src/config.rs

- Created: 2025-08-25 06:03:20 UTC | Link: https://github.com/openai/codex/pull/2651#discussion_r2297194678

```diff
@@ -485,6 +485,14 @@ pub struct ConfigToml {
 
     /// Nested tools section for feature toggles
     pub tools: Option<ToolsToml>,
+
+    /// Include an experimental plan tool that the model can use to update its
+    /// current plan and status of each step.
+    pub include_plan_tool: Option<bool>,
```

> Do you need to rebase this PR? This seems unrelated.

### codex-rs/core/src/tool_apply_patch.rs

- Created: 2025-08-25 06:06:44 UTC | Link: https://github.com/openai/codex/pull/2651#discussion_r2297199139

```diff
@@ -39,12 +39,12 @@ delete_hunk: "*** Delete File: " filename LF
 update_hunk: "*** Update File: " filename LF change_move? change?
 
 filename: /(.+)/
-add_line: "+" /(.+)/ LF -> line
+add_line: "+" /(.*)/ LF -> line
```

> I'm happy we caught the issue, though is there a unit test we could add for this?
> 
> It feels like it would probably have to be in Python since I don't see mature crates, in which case perhaps it is more trouble than it's worth?
> 
> Also, should this definition live in its own file (`.lark`? `.grammar`?) and then pull this in with `include!()`?