Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

超时报错问题 #15

Closed
ruibo5 opened this issue Oct 30, 2024 · 16 comments
Closed

超时报错问题 #15

ruibo5 opened this issue Oct 30, 2024 · 16 comments

Comments

@ruibo5
Copy link

ruibo5 commented Oct 30, 2024

你好 我调用了 zhipu ai 的免费 api. GLM-4-Flash
然后出现了以下超时报错, config 文件里面的 timeout 默认设定是 20 , 最开始报超市错误 我把timeout超参数设置成了600 但是依然报错， 请是否能帮助我解决一下这个问题。

(reevo-main) lab@lab-desktop:/ssd1/reevo-main$ python main.py problem=tsp_aco
[2024-10-30 16:03:32,051][root][INFO] - Workspace: /ssd1/SNU/reevo-main/outputs/tsp_aco-aco/2024-10-30_16-03-32
[2024-10-30 16:03:32,051][root][INFO] - Project Root: /ssd1/SNU/reevo-main
[2024-10-30 16:03:32,051][root][INFO] - Using LLM: GLM-4-Flash
[2024-10-30 16:03:32,051][root][INFO] - Using Algorithm: reevo
[2024-10-30 16:03:32,417][root][INFO] - Problem: tsp_aco
[2024-10-30 16:03:32,423][root][INFO] - Problem description: Solving Traveling Salesman Problem (TSP) via stochastic solution sampling following "heuristics". TSP requires finding the shortest path that visits all given nodes and returns to the starting node.
[2024-10-30 16:03:32,423][root][INFO] - Function name: heuristics
[2024-10-30 16:03:32,423][root][INFO] - Evaluating seed function...
[2024-10-30 16:03:32,423][root][INFO] - Seed function code:
import numpy as np
def heuristics_v2(distance_matrix: np.ndarray) -> np.ndarray:
return 1 / distance_matrix
[2024-10-30 16:03:32,423][root][INFO] - Iteration 0: Running Code 0
[2024-10-30 16:03:33,686][root][INFO] - Iteration 0: Code Run 0 successful!
[2024-10-30 16:03:40,360][root][INFO] - Iteration 0, response_id 0: Objective value: 6.627531378063073
[2024-10-30 16:03:40,361][root][INFO] - Iteration 0: Elitist: 6.627531378063073
[2024-10-30 16:03:40,361][root][INFO] - Iteration 0 finished...
[2024-10-30 16:03:40,361][root][INFO] - Best obj: 6.627531378063073, Best Code Path: problem_iter0_code0.py
[2024-10-30 16:03:40,361][root][INFO] - Function Evals: 1
[2024-10-30 16:03:40,361][root][INFO] - Initial Population Prompt:
System Prompt:
You are an expert in the domain of optimization heuristics. Your task is to design heuristics that can effectively solve optimization problems.
Your response outputs Python code and nothing else. Format your code as a Python code string: "python ... ".

User Prompt:
Write a heuristics function for Solving Traveling Salesman Problem (TSP) via stochastic solution sampling following "heuristics". TSP requires finding the shortest path that visits all given nodes and returns to the starting node.
The heuristics function takes as input a distance matrix, and returns prior indicators of how promising it is to include each edge in a solution. The return is of the same shape as the input.

def heuristics_v1(distance_matrix: np.ndarray) -> np.ndarray:
return 1 / distance_matrix

Refer to the format of a trivial design above. Be very creative and give heuristics_v2. Output code only and enclose your code with Python code block: python ... .

  • Try combining various factors to determine how promising it is to select an edge.
  • Try sparsifying the matrix by setting unpromising elements to zero.
    [2024-10-30 16:03:44,381][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
    [2024-10-30 16:03:45,224][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
    [2024-10-30 16:03:45,934][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
    [2024-10-30 16:03:46,429][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
    [2024-10-30 16:03:46,429][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
    [2024-10-30 16:03:47,178][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
    [2024-10-30 16:03:47,497][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
    [2024-10-30 16:03:47,661][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
    [2024-10-30 16:03:48,004][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
    [2024-10-30 16:03:48,377][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
    [2024-10-30 16:03:48,716][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
    [2024-10-30 16:03:48,722][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
    [2024-10-30 16:03:48,839][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
    [2024-10-30 16:03:48,905][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
    [2024-10-30 16:03:49,606][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
    [2024-10-30 16:03:49,713][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
    [2024-10-30 16:03:50,018][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
    [2024-10-30 16:03:50,624][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
    [2024-10-30 16:03:50,754][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
    [2024-10-30 16:03:51,039][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
    [2024-10-30 16:03:51,856][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
    [2024-10-30 16:03:51,937][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
    [2024-10-30 16:03:52,138][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
    [2024-10-30 16:03:52,769][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
    [2024-10-30 16:03:53,799][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
    [2024-10-30 16:03:54,413][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
    [2024-10-30 16:03:55,441][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
    [2024-10-30 16:03:56,717][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
    [2024-10-30 16:03:56,749][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
    [2024-10-30 16:03:59,717][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
    [2024-10-30 16:03:59,727][root][INFO] - Iteration 1: Running Code 0
    [2024-10-30 16:04:00,899][root][INFO] - Iteration 1: Code Run 0 successful!
    [2024-10-30 16:04:00,899][root][INFO] - Iteration 1: Running Code 1
    [2024-10-30 16:04:02,187][root][INFO] - Iteration 1: Code Run 1 successful!
    [2024-10-30 16:04:02,187][root][INFO] - Iteration 1: Running Code 2
    [2024-10-30 16:04:03,689][root][INFO] - Iteration 1: Code Run 2 successful!
    [2024-10-30 16:04:03,689][root][INFO] - Iteration 1: Running Code 3
    [2024-10-30 16:04:05,235][root][INFO] - Iteration 1: Code Run 3 successful!
    [2024-10-30 16:04:05,235][root][INFO] - Iteration 1: Running Code 4
    [2024-10-30 16:04:06,788][root][INFO] - Iteration 1: Code Run 4 successful!
    [2024-10-30 16:04:06,788][root][INFO] - Iteration 1: Running Code 5
    [2024-10-30 16:04:08,445][root][INFO] - Iteration 1: Code Run 5 successful!
    [2024-10-30 16:04:08,445][root][INFO] - Iteration 1: Running Code 6
    [2024-10-30 16:04:10,121][root][INFO] - Iteration 1: Code Run 6 successful!
    [2024-10-30 16:04:10,121][root][INFO] - Iteration 1: Running Code 7
    [2024-10-30 16:04:12,435][root][INFO] - Iteration 1: Code Run 7 successful!
    [2024-10-30 16:04:12,435][root][INFO] - Iteration 1: Running Code 8
    [2024-10-30 16:04:14,561][root][INFO] - Iteration 1: Code Run 8 successful!
    [2024-10-30 16:04:14,562][root][INFO] - Iteration 1: Running Code 9
    [2024-10-30 16:04:16,723][root][INFO] - Iteration 1: Code Run 9 successful!
    [2024-10-30 16:04:16,724][root][INFO] - Iteration 1: Running Code 10
    [2024-10-30 16:04:19,927][root][INFO] - Iteration 1: Code Run 10 execution error!
    [2024-10-30 16:04:19,927][root][INFO] - Iteration 1: Running Code 11
    [2024-10-30 16:04:22,127][root][INFO] - Iteration 1: Code Run 11 successful!
    [2024-10-30 16:04:22,127][root][INFO] - Iteration 1: Running Code 12
    [2024-10-30 16:04:25,174][root][INFO] - Iteration 1: Code Run 12 successful!
    [2024-10-30 16:04:25,181][root][INFO] - Iteration 1: Running Code 13
    [2024-10-30 16:04:27,708][root][INFO] - Iteration 1: Code Run 13 successful!
    [2024-10-30 16:04:27,708][root][INFO] - Iteration 1: Running Code 14
    [2024-10-30 16:04:29,989][root][INFO] - Iteration 1: Code Run 14 successful!
    [2024-10-30 16:04:29,990][root][INFO] - Iteration 1: Running Code 15
    [2024-10-30 16:04:32,990][root][INFO] - Iteration 1: Code Run 15 successful!
    [2024-10-30 16:04:32,990][root][INFO] - Iteration 1: Running Code 16
    [2024-10-30 16:04:35,998][root][INFO] - Iteration 1: Code Run 16 successful!
    [2024-10-30 16:04:35,999][root][INFO] - Iteration 1: Running Code 17
    [2024-10-30 16:04:40,786][root][INFO] - Iteration 1: Code Run 17 successful!
    [2024-10-30 16:04:40,787][root][INFO] - Iteration 1: Running Code 18
    [2024-10-30 16:04:49,578][root][INFO] - Iteration 1: Code Run 18 successful!
    [2024-10-30 16:04:49,578][root][INFO] - Iteration 1: Running Code 19
    [2024-10-30 16:04:55,430][root][INFO] - Iteration 1: Code Run 19 successful!
    [2024-10-30 16:04:55,431][root][INFO] - Iteration 1: Running Code 20
    [2024-10-30 16:05:05,903][root][INFO] - Iteration 1: Code Run 20 successful!
    [2024-10-30 16:05:05,903][root][INFO] - Iteration 1: Running Code 21
    [2024-10-30 16:05:10,701][root][INFO] - Iteration 1: Code Run 21 execution error!
    [2024-10-30 16:05:10,701][root][INFO] - Iteration 1: Running Code 22
    [2024-10-30 16:05:15,312][root][INFO] - Iteration 1: Code Run 22 successful!
    [2024-10-30 16:05:15,312][root][INFO] - Iteration 1: Running Code 23
    [2024-10-30 16:05:19,379][root][INFO] - Iteration 1: Code Run 23 successful!
    [2024-10-30 16:05:19,379][root][INFO] - Iteration 1: Running Code 24
    [2024-10-30 16:05:23,234][root][INFO] - Iteration 1: Code Run 24 successful!
    [2024-10-30 16:05:23,235][root][INFO] - Iteration 1: Running Code 25
    [2024-10-30 16:05:27,360][root][INFO] - Iteration 1: Code Run 25 successful!
    [2024-10-30 16:05:27,361][root][INFO] - Iteration 1: Running Code 26
    [2024-10-30 16:05:30,673][root][INFO] - Iteration 1: Code Run 26 execution error!
    [2024-10-30 16:05:30,676][root][INFO] - Iteration 1: Running Code 27
    [2024-10-30 16:05:32,799][root][INFO] - Iteration 1: Code Run 27 successful!
    [2024-10-30 16:05:32,799][root][INFO] - Iteration 1: Running Code 28
    [2024-10-30 16:05:35,549][root][INFO] - Iteration 1: Code Run 28 execution error!
    [2024-10-30 16:05:35,549][root][INFO] - Iteration 1: Running Code 29
    [2024-10-30 16:05:39,294][root][INFO] - Iteration 1: Code Run 29 successful!
    [2024-10-30 16:15:39,294][root][INFO] - Error for response_id 0: Command '['python', '-u', '/ssd1/SNU/reevo-main/problems/tsp_aco/eval.py', '50', '/ssd1/reevo-main', 'train']' timed out after 599.9999819584191 seconds
    ^C^CTraceback (most recent call last):
    File "/home/lab/anaconda3/envs/reevo-main/lib/python3.11/subprocess.py", line 1264, in wait
    return self._wait(timeout=timeout)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/lab/anaconda3/envs/reevo-main/lib/python3.11/subprocess.py", line 2047, in _wait
    time.sleep(delay)
    KeyboardInterrupt
@henry-yeh
Copy link
Member

您好，尝试一下更小的种群数量：
python main.py problem=tsp_aco init_pop_size=2 pop_size=2

查看下outputs文件夹下response.txt是否有问题（例如死循环）；查看下stdout.txt的输出内容，是否正常输出，例如：
[*] Instance 0: 6.429725805964564

@ruibo5
Copy link
Author

ruibo5 commented Oct 30, 2024

你好。我按照您给的指令输入进去跑了一下 ， 还是报错。 错误 和 stdout 给您附上：

(reevo-main) lab@lab-desktop:/ssd1/SNU/reevo-main$ python main.py problem=tsp_aco init_pop_size=2 pop_size=2
[2024-10-30 19:30:22,580][root][INFO] - Workspace: /ssd1/SNU/reevo-main/outputs/tsp_aco-aco/2024-10-30_19-30-22
[2024-10-30 19:30:22,580][root][INFO] - Project Root: /ssd1/SNU/reevo-main
[2024-10-30 19:30:22,580][root][INFO] - Using LLM: GLM-4-Flash
[2024-10-30 19:30:22,580][root][INFO] - Using Algorithm: reevo
[2024-10-30 19:30:22,927][root][INFO] - Problem: tsp_aco
[2024-10-30 19:30:22,927][root][INFO] - Problem description: Solving Traveling Salesman Problem (TSP) via stochastic solution sampling following "heuristics". TSP requires finding the shortest path that visits all given nodes and returns to the starting node.
[2024-10-30 19:30:22,928][root][INFO] - Function name: heuristics
[2024-10-30 19:30:22,928][root][INFO] - Evaluating seed function...
[2024-10-30 19:30:22,928][root][INFO] - Seed function code:
import numpy as np
def heuristics_v2(distance_matrix: np.ndarray) -> np.ndarray:
return 1 / distance_matrix
[2024-10-30 19:30:22,929][root][INFO] - Iteration 0: Running Code 0
[2024-10-30 19:30:24,148][root][INFO] - Iteration 0: Code Run 0 successful!
[2024-10-30 19:30:29,873][root][INFO] - Iteration 0, response_id 0: Objective value: 6.583946489201288
[2024-10-30 19:30:29,873][root][INFO] - Iteration 0: Elitist: 6.583946489201288
[2024-10-30 19:30:29,873][root][INFO] - Iteration 0 finished...
[2024-10-30 19:30:29,873][root][INFO] - Best obj: 6.583946489201288, Best Code Path: problem_iter0_code0.py
[2024-10-30 19:30:29,873][root][INFO] - Function Evals: 1
[2024-10-30 19:30:29,873][root][INFO] - Initial Population Prompt:
System Prompt:
You are an expert in the domain of optimization heuristics. Your task is to design heuristics that can effectively solve optimization problems.
Your response outputs Python code and nothing else. Format your code as a Python code string: "python ... ".

User Prompt:
Write a heuristics function for Solving Traveling Salesman Problem (TSP) via stochastic solution sampling following "heuristics". TSP requires finding the shortest path that visits all given nodes and returns to the starting node.
The heuristics function takes as input a distance matrix, and returns prior indicators of how promising it is to include each edge in a solution. The return is of the same shape as the input.

def heuristics_v1(distance_matrix: np.ndarray) -> np.ndarray:
return 1 / distance_matrix

Refer to the format of a trivial design above. Be very creative and give heuristics_v2. Output code only and enclose your code with Python code block: python ... .

  • Try combining various factors to determine how promising it is to select an edge.
  • Try sparsifying the matrix by setting unpromising elements to zero.
    [2024-10-30 19:30:45,711][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
    [2024-10-30 19:30:50,770][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
    [2024-10-30 19:30:50,776][root][INFO] - Iteration 1: Running Code 0
    [2024-10-30 19:30:51,952][root][INFO] - Iteration 1: Code Run 0 successful!
    [2024-10-30 19:30:51,952][root][INFO] - Iteration 1: Running Code 1
    [2024-10-30 19:30:53,222][root][INFO] - Iteration 1: Code Run 1 successful!
    [2024-10-30 19:30:58,044][root][INFO] - Iteration 1, response_id 0: Objective value: 20.647744666060657
    [2024-10-30 19:30:58,044][root][INFO] - Iteration 1, response_id 1: Objective value: inf
    [2024-10-30 19:30:58,044][root][INFO] - Iteration 1 finished...
    [2024-10-30 19:30:58,044][root][INFO] - Best obj: 6.583946489201288, Best Code Path: problem_iter0_code0.py
    [2024-10-30 19:30:58,045][root][INFO] - Function Evals: 3
    [2024-10-30 19:30:58,045][root][INFO] - Short-term Reflection Prompt:
    System Prompt:
    You are an expert in the domain of optimization heuristics. Your task is to give hints to design better heuristics.

User Prompt:
Below are two heuristics functions for Solving Traveling Salesman Problem (TSP) via stochastic solution sampling following "heuristics". TSP requires finding the shortest path that visits all given nodes and returns to the starting node.
The heuristics function takes as input a distance matrix, and returns prior indicators of how promising it is to include each edge in a solution. The return is of the same shape as the input.

You are provided with two code versions below, where the second version performs better than the first one.

[Worse code]

# Create a matrix with the sum of each row (closeness to all other nodes)
row_sums = np.sum(distance_matrix, axis=1)
# Normalize row sums to get a probability distribution
probability_distribution = row_sums / np.sum(row_sums)
# Create a matrix where each element is a weighted sum of its own value and the inverse of its row sum
weighted_distance = distance_matrix * (probability_distribution[:, np.newaxis])
# Create a threshold based on the maximum weighted distance
threshold = np.max(weighted_distance)
# Sparsify the matrix by setting elements below the threshold to zero
sparsified_matrix = np.where(weighted_distance < threshold, 0, weighted_distance)
# Return the sparsified matrix as the heuristic indicator
return sparsified_matrix

[Better code]
return 1 / distance_matrix

You respond with some hints for designing better heuristics, based on the two code versions and using less than 20 words.
[2024-10-30 19:31:01,231][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
[2024-10-30 19:31:01,553][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
[2024-10-30 19:31:01,557][root][INFO] - Crossover Prompt:
System Prompt:
You are an expert in the domain of optimization heuristics. Your task is to design heuristics that can effectively solve optimization problems.
Your response outputs Python code and nothing else. Format your code as a Python code string: "python ... ".

User Prompt:
Write a heuristics function for Solving Traveling Salesman Problem (TSP) via stochastic solution sampling following "heuristics". TSP requires finding the shortest path that visits all given nodes and returns to the starting node.
The heuristics function takes as input a distance matrix, and returns prior indicators of how promising it is to include each edge in a solution. The return is of the same shape as the input.

[Worse code]
def heuristics_v0(distance_matrix: np.ndarray) -> np.ndarray:

# Create a matrix with the sum of each row (closeness to all other nodes)
row_sums = np.sum(distance_matrix, axis=1)
# Normalize row sums to get a probability distribution
probability_distribution = row_sums / np.sum(row_sums)
# Create a matrix where each element is a weighted sum of its own value and the inverse of its row sum
weighted_distance = distance_matrix * (probability_distribution[:, np.newaxis])
# Create a threshold based on the maximum weighted distance
threshold = np.max(weighted_distance)
# Sparsify the matrix by setting elements below the threshold to zero
sparsified_matrix = np.where(weighted_distance < threshold, 0, weighted_distance)
# Return the sparsified matrix as the heuristic indicator
return sparsified_matrix

[Better code]
def heuristics_v1(distance_matrix: np.ndarray) -> np.ndarray:
return 1 / distance_matrix

[Reflection]
Focus on information content, avoid redundant operations, and prioritize simpler computations.

[Improved code]
Please write an improved function heuristics_v2, according to the reflection. Output code only and enclose your code with Python code block: python ... .
[2024-10-30 19:31:05,513][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
[2024-10-30 19:31:09,677][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
[2024-10-30 19:31:09,680][root][INFO] - Iteration 2: Running Code 0
[2024-10-30 19:31:10,897][root][INFO] - Iteration 2: Code Run 0 successful!
[2024-10-30 19:31:10,898][root][INFO] - Iteration 2: Running Code 1
[2024-10-30 19:31:12,234][root][INFO] - Iteration 2: Code Run 1 successful!
[2024-10-30 19:31:26,502][root][INFO] - Iteration 2, response_id 0: Objective value: 6.539949609732332
[2024-10-30 19:31:26,502][root][INFO] - Iteration 2, response_id 1: Objective value: 6.601617118843419
[2024-10-30 19:31:26,502][root][INFO] - Iteration 2: Elitist: 6.539949609732332
[2024-10-30 19:31:26,502][root][INFO] - Iteration 2 finished...
[2024-10-30 19:31:26,502][root][INFO] - Best obj: 6.539949609732332, Best Code Path: problem_iter2_code0.py
[2024-10-30 19:31:26,502][root][INFO] - Function Evals: 5
[2024-10-30 19:31:26,502][root][INFO] - Long-term Reflection Prompt:
System Prompt:
You are an expert in the domain of optimization heuristics. Your task is to give hints to design better heuristics.

User Prompt:
Below is your prior long-term reflection on designing heuristics for Solving Traveling Salesman Problem (TSP) via stochastic solution sampling following "heuristics". TSP requires finding the shortest path that visits all given nodes and returns to the starting node.

  • Try combining various factors to determine how promising it is to select an edge.
  • Try sparsifying the matrix by setting unpromising elements to zero.

Below are some newly gained insights.
Focus on information content, avoid redundant operations, and prioritize simpler computations.
Consider relative distances, not absolute, and use common sense; simpler often beats complex.

Write constructive hints for designing better heuristics, based on prior reflections and new insights and using less than 50 words.
[2024-10-30 19:31:32,743][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
[2024-10-30 19:31:32,745][root][INFO] - Mutation Prompt:
System Prompt:
You are an expert in the domain of optimization heuristics. Your task is to design heuristics that can effectively solve optimization problems.
Your response outputs Python code and nothing else. Format your code as a Python code string: "python ... ".

User Prompt:
Write a heuristics function for Solving Traveling Salesman Problem (TSP) via stochastic solution sampling following "heuristics". TSP requires finding the shortest path that visits all given nodes and returns to the starting node.
The heuristics function takes as input a distance matrix, and returns prior indicators of how promising it is to include each edge in a solution. The return is of the same shape as the input.

[Prior reflection]

  1. Evaluate edge selection with a weighted sum of factors.
  2. Apply matrix sparsity to filter out unpromising paths.
  3. Prioritize informative choices over complexity.
  4. Use relative distances and common sense over intricate metrics.- Try combining various factors to determine how promising it is to select an edge.
  • Try sparsifying the matrix by setting unpromising elements to zero.

[Code]
def heuristics_v1(distance_matrix: np.ndarray) -> np.ndarray:

return 1 / (distance_matrix + np.finfo(float).eps)

[Improved code]
Please write a mutated function heuristics_v2, according to the reflection. Output code only and enclose your code with Python code block: python ... .
[2024-10-30 19:31:40,591][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
[2024-10-30 19:31:40,593][root][INFO] - Iteration 3: Running Code 0
[2024-10-30 19:31:41,800][root][INFO] - Iteration 3: Code Run 0 successful!
[2024-10-30 19:31:47,622][root][INFO] - Iteration 3, response_id 0: Objective value: 6.548594326308752
[2024-10-30 19:31:47,622][root][INFO] - Iteration 3 finished...
[2024-10-30 19:31:47,622][root][INFO] - Best obj: 6.539949609732332, Best Code Path: problem_iter2_code0.py
[2024-10-30 19:31:47,622][root][INFO] - Function Evals: 6
[2024-10-30 19:31:50,194][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
[2024-10-30 19:31:51,107][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
[2024-10-30 19:32:05,239][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
[2024-10-30 19:32:05,241][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
[2024-10-30 19:32:05,244][root][INFO] - Iteration 4: Running Code 0
[2024-10-30 19:32:06,453][root][INFO] - Iteration 4: Code Run 0 successful!
[2024-10-30 19:32:06,453][root][INFO] - Iteration 4: Running Code 1
[2024-10-30 19:32:07,648][root][INFO] - Iteration 4: Code Run 1 successful!
[2024-10-30 19:32:07,648][root][INFO] - Iteration 4, response_id 0: Objective value: inf
[2024-10-30 19:32:13,670][root][INFO] - Iteration 4, response_id 1: Objective value: 5.998063864749301
[2024-10-30 19:32:13,670][root][INFO] - Iteration 4: Elitist: 5.998063864749301
[2024-10-30 19:32:13,670][root][INFO] - Iteration 4 finished...
[2024-10-30 19:32:13,670][root][INFO] - Best obj: 5.998063864749301, Best Code Path: problem_iter4_code1.py
[2024-10-30 19:32:13,670][root][INFO] - Function Evals: 8
[2024-10-30 19:32:18,155][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
[2024-10-30 19:32:58,379][httpx][INFO] - HTTP Request: POST https://open.bigmodel.cn/api/paas/v4/chat/completions "HTTP/1.1 200 OK"
[2024-10-30 19:32:58,382][root][INFO] - Iteration 5: Running Code 0
[2024-10-30 19:32:59,553][root][INFO] - Iteration 5: Code Run 0 successful!
[2024-10-30 19:32:59,867][root][INFO] - Iteration 5, response_id 0: Objective value: inf
[2024-10-30 19:32:59,867][root][INFO] - Iteration 5 finished...
[2024-10-30 19:32:59,867][root][INFO] - Best obj: 5.998063864749301, Best Code Path: problem_iter4_code1.py
[2024-10-30 19:32:59,867][root][INFO] - Function Evals: 9
Error executing job with overrides: ['problem=tsp_aco', 'init_pop_size=2', 'pop_size=2']
Traceback (most recent call last):
File "/ssd1/SNU/reevo-main/main.py", line 54, in
main()
File "/home/lab/anaconda3/envs/reevo-main/lib/python3.11/site-packages/hydra/main.py", line 94, in decorated_main
_run_hydra(
File "/home/lab/anaconda3/envs/reevo-main/lib/python3.11/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
_run_app(
File "/home/lab/anaconda3/envs/reevo-main/lib/python3.11/site-packages/hydra/_internal/utils.py", line 457, in _run_app
run_and_report(
File "/home/lab/anaconda3/envs/reevo-main/lib/python3.11/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
raise ex
File "/home/lab/anaconda3/envs/reevo-main/lib/python3.11/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
return func()
^^^^^^
File "/home/lab/anaconda3/envs/reevo-main/lib/python3.11/site-packages/hydra/_internal/utils.py", line 458, in
lambda: hydra.run(
^^^^^^^^^^
File "/home/lab/anaconda3/envs/reevo-main/lib/python3.11/site-packages/hydra/_internal/hydra.py", line 132, in run
_ = ret.return_value
^^^^^^^^^^^^^^^^
File "/home/lab/anaconda3/envs/reevo-main/lib/python3.11/site-packages/hydra/core/utils.py", line 260, in return_value
raise self._return_value
File "/home/lab/anaconda3/envs/reevo-main/lib/python3.11/site-packages/hydra/core/utils.py", line 186, in run_job
ret.return_value = task_function(task_cfg)
^^^^^^^^^^^^^^^^^^^^^^^
File "/ssd1/SNU/reevo-main/main.py", line 34, in main
best_code_overall, best_code_path_overall = lhh.evolve()
^^^^^^^^^^^^
File "/ssd1/SNU/reevo-main/reevo.py", line 456, in evolve
raise RuntimeError("Selection failed. Please check the population.")
RuntimeError: Selection failed. Please check the population.

stdout:
[] Running ...
[] Dataset loaded: /ssd1/SNU/reevo-main/problems/tsp_aco/dataset/train50_dataset.npy with 5 instances.
/ssd1/SNU/reevo-main/problems/tsp_aco/gpt.py:21: RuntimeWarning: invalid value encountered in divide
normalized_factor = combined_factor / np.max(combined_factor)
Traceback (most recent call last):
File "/ssd1/SNU/reevo-main/problems/tsp_aco/eval.py", line 53, in
obj = solve(node_pos)
^^^^^^^^^^^^^^^
File "/ssd1/SNU/reevo-main/problems/tsp_aco/eval.py", line 29, in solve
obj = aco.run(N_ITERATIONS)
^^^^^^^^^^^^^^^^^^^^^
File "/home/lab/anaconda3/envs/reevo-main/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/ssd1/SNU/reevo-main/problems/tsp_aco/aco.py", line 34, in run
paths = self.gen_path(require_prob=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/ssd1/SNU/reevo-main/problems/tsp_aco/aco.py", line 92, in gen_path
actions, log_probs = self.pick_move(prev, mask, require_prob)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/ssd1/SNU/reevo-main/problems/tsp_aco/aco.py", line 114, in pick_move
dist = Categorical(dist)
^^^^^^^^^^^^^^^^^
File "/home/lab/anaconda3/envs/reevo-main/lib/python3.11/site-packages/torch/distributions/categorical.py", line 72, in init
super().init(batch_shape, validate_args=validate_args)
File "/home/lab/anaconda3/envs/reevo-main/lib/python3.11/site-packages/torch/distributions/distribution.py", line 71, in init
raise ValueError(
ValueError: Expected parameter probs (Tensor of shape (30, 50)) of distribution Categorical(probs: torch.Size([30, 50])) to satisfy the constraint Simplex(), but found invalid values:
tensor([[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
...,
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan]], dtype=torch.float64)

@henry-yeh
Copy link
Member

此次运行没有问题，此处报错是因为我们将population size设置为2进行测试，出现了错误的heuristic后（这是因为LLM生成的问题），selection会失败。将population size调大即可避免这次报错。

由于减小population size即可顺利运行，您之前的问题是因为硬件性能。可以采取较小的init_population_size，或修改代码单进程评估population。

@henry-yeh
Copy link
Member

会影响的

@ruibo5
Copy link
Author

ruibo5 commented Oct 30, 2024

您好 init_population_size 设置太小 例如init_population_size=5 会影响性能吗, 还有就是 论文里面用的是chatgpt, 而如果我用 GLM-4-Flash 会影响性能吗

@henry-yeh
Copy link
Member

采用不同参数会影响结果复现，对于新的应用影响不会太大

@ruibo5
Copy link
Author

ruibo5 commented Oct 31, 2024

你好 请问这个tsp_aco 20 , 50 100个点 , 数据集是在哪里找的， 如果是自己手动创建, 该如何创建?

@henry-yeh
Copy link
Member

参见代码：https://github.com/ai4co/reevo/blob/main/problems/tsp_aco/gen_inst.py

@ruibo5
Copy link
Author

ruibo5 commented Nov 1, 2024

您好我有以下两个问题想问一下您

1.请问这两个路径下的（ problems/tsp_aco/test.ipynb 和 problems/tsp_gls/test.ipynb）的代码def heuristics_reevo(distance_matrix: np.ndarray) 为什么不一样

第1个问题的具体代码内容如下
1.
在problems/tsp_aco/test.ipynb路径下的代码为以下：
def heuristics_reevo(edge_attr):
num_edges = edge_attr.shape[0]
num_attributes = edge_attr.shape[1]

heuristic_values = np.zeros_like(edge_attr)

# Apply feature engineering on edge attributes
transformed_attr = np.log1p(np.abs(edge_attr))  # Taking logarithm of absolute value of attributes

# Normalize edge attributes
scaler = StandardScaler()
edge_attr_norm = scaler.fit_transform(transformed_attr)

# Calculate correlation coefficients
correlation_matrix = np.corrcoef(edge_attr_norm.T)

# Calculate heuristic value for each edge attribute
for i in range(num_edges):
    for j in range(num_attributes):
        if edge_attr_norm[i][j] != 0:
            heuristic_values[i][j] = np.exp(-8 * edge_attr_norm[i][j] * correlation_matrix[j][j])

return heuristic_values

在problems/tsp_gls/test.ipynb 路径下的代码为以下:

def heuristics_reevo(distance_matrix: np.ndarray) -> np.ndarray:
# Calculate the average distance for each node
average_distance = np.mean(distance_matrix, axis=1)

# Calculate the distance ranking for each node
distance_ranking = np.argsort(distance_matrix, axis=1)

# Calculate the mean of the closest distances for each node
closest_mean_distance = np.mean(distance_matrix[np.arange(distance_matrix.shape[0])[:, None], distance_ranking[:, 1:5]], axis=1)

# Initialize the indicator matrix and calculate ratio of distance to average distance
indicators = distance_matrix / average_distance[:, np.newaxis]

# Set diagonal elements to np.inf
np.fill_diagonal(indicators, np.inf)

# Adjust the indicator matrix using the statistical measure
indicators += closest_mean_distance[:, np.newaxis] / np.sum(distance_matrix, axis=1)[:, np.newaxis]

return indicators

def vanilla_ktsp(distance_matrix: np.ndarray) -> np.ndarray:
return distance_matrix

2-1.problems/tsp_gls/test.ipynb 里面的代码optimality gap 我跑出来了负数 是可以看作是0吗
2-2. 这个代码是否是论文中Table1 里面的 KGLS + ReEvo(ours) 方法, 在Table1 里面 Tsp 200 的opt gap 是 0.216
然后在Table1里面其他的 Tsp 20 50 100 opt gap 全是 0

请问 我在problems/tsp_gls/test.ipynb代码文件里面跑出来的结果是否是是您Table1 里面KGLS + ReEvo(ours) 方法的结果
。我跑出来的以下两个evaluate(heuristics_reevo), evaluate(vanilla_ktsp) 如果不是的话是什么结果？

evaluate(heuristics_reevo)

erage for 20: 3.746594 (3.836285)
[] Optimality gap: -2.337967%
[] Total/Average duration: 0.064194s 0.001003s
[] Average for 50: 5.635503 (5.684580)
[] Optimality gap: -0.863328%
[] Total/Average duration: 1.737566s 0.027149s
[] Average for 100: 7.739454 (7.778580)
[] Optimality gap: -0.503004%
[] Total/Average duration: 80.867855s 1.263560s
[] Average for 200: 10.719096 (10.711946)
[] Optimality gap: 0.066744%
[*] Total/Average duration: 134.167998s 2.096375s

evaluate(vanilla_ktsp)

[] Average for 20: 3.746594 (3.836285)
[] Optimality gap: -2.337967%
[*] Total/Average duration: 0.051683s 0.000808s

[] Average for 50: 5.635709 (5.684580)
[] Optimality gap: -0.859714%
[*] Total/Average duration: 1.691789s 0.026434s

[] Average for 100: 7.739454 (7.778580)
[] Optimality gap: -0.503004%
[*] Total/Average duration: 82.049456s 1.282023s

[] Average for 200: 10.725571 (10.711946)
[] Optimality gap: 0.127199%
[*] Total/Average duration: 132.682913s 2.073171s

@henry-yeh
Copy link
Member

  1. 对不同问题设定，ReEvo分别训练heuristic，故不同。
    2.1 是因为没有跑出最优解。
    2.2 是文中结果，但是Optimality gap偏低，因为没有正确求得最优解。

@ruibo5
Copy link
Author

ruibo5 commented Nov 3, 2024

您好 有以下两个问题想问您
1.problems/tsp_gls/test.ipynb 我跑了 tsp_gls 问题的性能, 用您生成的代码测试的时候是和我自己跑的之后测试的性能不太一样, 用您生成的代码测试性能的时候和论文的Table1是一致的, 但是我用自己跑完之后生成的代码 性能和论文里的Table1有所差距.
我的 初始化种群设定为5 然后用的 GLM-4-Flash 语言模型. 我不知道达不到Table1性能的原因？ 请问如果想达到论文的性能该怎么做？

  1. 代码文件(problems/tsp_gls/test.ipynb)里的evaluate(heuristics_reevo) 是Table 1图标的性能测试代码, 请问evaluate(vanilla_ktsp)是测试什么的？

我的代码的性能：

evaluate(heuristics_v2)

[] Average for 20: 3.836448 (3.836285)
[] Optimality gap: 0.004242%
[*] Total/Average duration: 0.064002s 0.001000s

[] Average for 50: 5.685814 (5.684580)
[] Optimality gap: 0.021709%
[*] Total/Average duration: 1.726301s 0.026973s

[] Average for 100: 7.779085 (7.778580)
[] Optimality gap: 0.006484%
[*] Total/Average duration: 82.156380s 1.283693s

[] Average for 200: 10.746896 (10.711946)
[] Optimality gap: 0.326271%
[*] Total/Average duration: 136.174024s 2.127719s

evaluate(vanilla_ktsp)
[] Average for 20: 3.836448 (3.836285)
[] Optimality gap: 0.004242%
[*] Total/Average duration: 0.053119s 0.000830s

[] Average for 50: 5.685538 (5.684580)
[] Optimality gap: 0.016862%
[*] Total/Average duration: 1.664141s 0.026002s

[] Average for 100: 7.778761 (7.778580)
[] Optimality gap: 0.002327%
[*] Total/Average duration: 81.678915s 1.276233s

[] Average for 200: 10.742373 (10.711946)
[] Optimality gap: 0.284046%
[*] Total/Average duration: 134.198511s 2.096852s

官方代码的性能

evaluate(heuristics_reevo)
[] Average for 20: 3.836285 (3.836285)
[] Optimality gap: 0.000000%
[*] Total/Average duration: 0.062750s 0.000980s

[] Average for 50: 5.684580 (5.684580)
[] Optimality gap: 0.000000%
[*] Total/Average duration: 1.695302s 0.026489s

[] Average for 100: 7.778580 (7.778580)
[] Optimality gap: 0.000000%
[*] Total/Average duration: 81.078894s 1.266858s

[] Average for 200: 10.735125 (10.711946)
[] Optimality gap: 0.216382%
[*] Total/Average duration: 133.999463s 2.093742s

evaluate(vanilla_ktsp)

[] Average for 20: 3.836448 (3.836285)
[] Optimality gap: 0.004242%
[*] Total/Average duration: 0.055982s 0.000875s

[] Average for 50: 5.685538 (5.684580)
[] Optimality gap: 0.016862%
[*] Total/Average duration: 1.673721s 0.026152s

[] Average for 100: 7.778761 (7.778580)
[] Optimality gap: 0.002327%
[*] Total/Average duration: 80.711671s 1.261120s

[] Average for 200: 10.742373 (10.711946)
[] Optimality gap: 0.284046%
[*] Total/Average duration: 132.459149s 2.069674s

@henry-yeh henry-yeh reopened this Nov 4, 2024
@henry-yeh
Copy link
Member

  1. 请使用原文超参数，详见Table 8；遵照原文实验设定，详见Appendix C。
  2. 原文表1中的KGLS。

@ruibo5
Copy link
Author

ruibo5 commented Nov 5, 2024

您好, 论文中Table 9 是 tsp-gls随着 TSP20/50/100/200 的 Perturbation moves 和 Number of iterations 超参数设定
请问如果我要跑300-1000的话 该如何设定， 这两个超参数会带来怎样的影响, 怎么调这两个超参数？？

@henry-yeh
Copy link
Member

henry-yeh commented Nov 6, 2024 •
edited
Loading

通常更大规模问题需要更多Perturbation moves跳出局部最优，Number of iteration影响运行时间和求解效果。

@ruibo5
Copy link
Author

ruibo5 commented Nov 6, 2024

Number of iteration 请问您是怎么找的数值

@henry-yeh
Copy link
Member

基于问题规模、计算资源、理想训练时间、推理时间和求解质量的权衡，综合确定

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ruibo5 @henry-yeh